Skip to main content

Remove first n and last n character in huge file

JSON file often only in 1 line.
To remove first n character:
sed -i.bak -r '1s/^.{10}//' file
https://stackoverflow.com/questions/26400118/delete-first-n-characters-of-a-very-large-file-in-unix-shell

See first 50 bytes of file:
head -c 50 file

See last 50 bytes:
tail -c 50 file

remove last n character
sed '$ s/.$//' somefile  # slow and not work, don't know why
truncate -s-1 file # super fast
https://stackoverflow.com/questions/27305177/how-can-i-remove-the-last-character-of-a-file-in-unix

I use git-bash windows 10, JSON ab 3GB

Why not sed

The problem with a sed solution like sed '$ s/.$//' file is that it reads the whole file first (taking a long time with large files), then you need a temporary file (of the same size as the original):

One more time my psychological knowledge suck. The "right" answer here is not marked right, OK but it seem better.

Comments

Popular posts from this blog

AWS Elasticache Memcached connection

https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/accessing-elasticache.html#access-from-outside-aws http://hourlyapps.blogspot.com/2010/06/examples-of-memcached-commands.html Access memcached https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/GettingStarted.AuthorizeAccess.html Zip include hidden file https://stackoverflow.com/questions/12493206/zip-including-hidden-files phpmemcachedadmin ~ phpMyAdmin or phpPgAdmin ... telnet mycachecluster.eaogs8.0001.usw2.cache.amazonaws.com 11211 stats items stats cachedump 27 100 https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/VPCs.EC.html https://lzone.de/cheat-sheet/memcached VPC ID Security Group ID (sg-...) Cluster: The identifier for the cluster memcached1 Creation Time: The time (UTC) when the cluster was created January 9, 2019 at 11:47:16 AM UTC+7 Configuration Endpoint: The configuration endpoint of the cluster memcached1.ahgofe.cfg.usw1.cache.amazonaws.com:11211 St...

Simulate Fail2ban on Apache request spam with mod_evasive limitipconn ...

https://en.wikipedia.org/wiki/Manchu_alphabet https://en.wikipedia.org/wiki/Sweet_potato https://en.wikipedia.org/wiki/New_World_crops https://www.mdpi.com/journal/energies http://www.cired.net/publications/cired2007/pdfs/CIRED2007_0342_paper.pdf https://www.davidpashley.com/articles/writing-robust-shell-scripts/ trap command https://en.wikipedia.org/wiki/Race_condition https://unix.stackexchange.com/questions/172541/why-does-exit-1-not-exit-the-script exit 1 not work it seem { } brace bound fixed it. cat access_log | cut -d ' ' -f 1 > ip1 sort -n -t. -k1,1 -k2,2 -k3,3 -k4,4 | uniq -c | sort -n -r -s https://unix.stackexchange.com/questions/246104/unix-count-unique-ip-addresses-sort-them-by-most-frequent-and-also-sort-them https://stackoverflow.com/questions/20164696/how-to-block-spam-and-spam-bots-for-good-with-htaccess  Code: ------------------------------------------------------------------- #Block Spam Bots and Spam on your website #Block proxies...

Notes Windows 10 Virtualbox config, PHP Storm Japanese, custom PHP, Apache build, Postgresql

 cmd => Ctrl + Shift + Enter mklink "C:\Users\HauNT\Videos\host3" "C:\Windows\System32\drivers\etc\hosts" https://www.quora.com/How-to-create-a-router-in-php https://serverfault.com/questions/225155/virtualbox-how-to-set-up-networking-so-both-host-and-guest-can-access-internet 1 NAT + 1 host only config https://unix.stackexchange.com/questions/115464/how-to-properly-set-up-2-network-interfaces-in-centos-running-in-virtualbox DEVICE=eth0 TYPE=Ethernet #BOOTPROTO=dhcp BOOTPROTO=none #IPADDR=10.9.11.246 #PREFIX=24 #GATEWAY=10.9.11.1 #IPV4_FAILURE_FATAL=yes #HWADDR=08:00:27:CC:AC:AC ONBOOT=yes NAME="System eth0" [root@localhost www]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 # Advanced Micro Devices, Inc. [AMD] 79c970 [PCnet32 LANCE] DEVICE=eth1 IPADDR=192.168.56.28 <= no eff => auto like DHCP #GATEWAY=192.168.56.1 #BOOTPROTO=dhcp BOOTPROTO=static <= no eff ONBOOT=yes HWADDR=08:00:27:b4:20:10 [root@localhost www]# ...