Skip to main content

MySql remove duplicate on composite 2 or more columns

I've been write SQL to remove duplicate rows in PostgreSQL and many other trick with huge DB.

Now I try with MySQL.

Find all duplicate rows by composite columns, ie. user_id and file_date. Both is integer: file_date in unix_timestamp.

select   id,
 user_id,
         file_date,
         count(*)
from     detailed_claims
group by user_id,
         file_date
having   count(*) > 1

We just need id column to remove, so only select id:

select id from (select id, user_id, file_date, count(*) from detailed_claims group by user_id, file_date having count(*) > 1 ) as blondie

Result:
|id          |
|  46400 |
|  46421 |
|  46402 |
|  46159 |
|  46414 |
|  46157 |
|  46161 |
|  46163 |
|  46405 |
|  46164 |
|  46165 |
|  46166 |
|  46158 |
|  46167 |
|  46162 |
|  46406 |
|  46417 |
|  46177 |
+--------+
4885 rows in set (1.14 sec)

Now remove:

delete from detailed_claims where id in (select id from (select   id,
    user_id,
         file_date,
         count(*)
from     detailed_claims
group by user_id,
         file_date
having   count(*) > 1
                ) as blondie
)


OK, now we can add Unique indexes to 2 columns user_id and file_date without error msg.


Comments

Popular posts from this blog

AWS Elasticache Memcached connection

https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/accessing-elasticache.html#access-from-outside-aws http://hourlyapps.blogspot.com/2010/06/examples-of-memcached-commands.html Access memcached https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/GettingStarted.AuthorizeAccess.html Zip include hidden file https://stackoverflow.com/questions/12493206/zip-including-hidden-files phpmemcachedadmin ~ phpMyAdmin or phpPgAdmin ... telnet mycachecluster.eaogs8.0001.usw2.cache.amazonaws.com 11211 stats items stats cachedump 27 100 https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/VPCs.EC.html https://lzone.de/cheat-sheet/memcached VPC ID Security Group ID (sg-...) Cluster: The identifier for the cluster memcached1 Creation Time: The time (UTC) when the cluster was created January 9, 2019 at 11:47:16 AM UTC+7 Configuration Endpoint: The configuration endpoint of the cluster memcached1.ahgofe.cfg.usw1.cache.amazonaws.com:11211 St...

Simulate Fail2ban on Apache request spam with mod_evasive limitipconn ...

https://en.wikipedia.org/wiki/Manchu_alphabet https://en.wikipedia.org/wiki/Sweet_potato https://en.wikipedia.org/wiki/New_World_crops https://www.mdpi.com/journal/energies http://www.cired.net/publications/cired2007/pdfs/CIRED2007_0342_paper.pdf https://www.davidpashley.com/articles/writing-robust-shell-scripts/ trap command https://en.wikipedia.org/wiki/Race_condition https://unix.stackexchange.com/questions/172541/why-does-exit-1-not-exit-the-script exit 1 not work it seem { } brace bound fixed it. cat access_log | cut -d ' ' -f 1 > ip1 sort -n -t. -k1,1 -k2,2 -k3,3 -k4,4 | uniq -c | sort -n -r -s https://unix.stackexchange.com/questions/246104/unix-count-unique-ip-addresses-sort-them-by-most-frequent-and-also-sort-them https://stackoverflow.com/questions/20164696/how-to-block-spam-and-spam-bots-for-good-with-htaccess  Code: ------------------------------------------------------------------- #Block Spam Bots and Spam on your website #Block proxies...

Notes Windows 10 Virtualbox config, PHP Storm Japanese, custom PHP, Apache build, Postgresql

 cmd => Ctrl + Shift + Enter mklink "C:\Users\HauNT\Videos\host3" "C:\Windows\System32\drivers\etc\hosts" https://www.quora.com/How-to-create-a-router-in-php https://serverfault.com/questions/225155/virtualbox-how-to-set-up-networking-so-both-host-and-guest-can-access-internet 1 NAT + 1 host only config https://unix.stackexchange.com/questions/115464/how-to-properly-set-up-2-network-interfaces-in-centos-running-in-virtualbox DEVICE=eth0 TYPE=Ethernet #BOOTPROTO=dhcp BOOTPROTO=none #IPADDR=10.9.11.246 #PREFIX=24 #GATEWAY=10.9.11.1 #IPV4_FAILURE_FATAL=yes #HWADDR=08:00:27:CC:AC:AC ONBOOT=yes NAME="System eth0" [root@localhost www]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 # Advanced Micro Devices, Inc. [AMD] 79c970 [PCnet32 LANCE] DEVICE=eth1 IPADDR=192.168.56.28 <= no eff => auto like DHCP #GATEWAY=192.168.56.1 #BOOTPROTO=dhcp BOOTPROTO=static <= no eff ONBOOT=yes HWADDR=08:00:27:b4:20:10 [root@localhost www]# ...