Bash Resources - selmling/Analytics-and-Data-Exploration GitHub Wiki

1. convert xlsx to csv:

  for /r %%v in (*.xlsx) do ssconvert "%%v" "%%vx.csv"

2. compile all .csvs (includes file name in column A):

grep -H "[A-Za-z0-9]" *.csv | cut -d, -f1-5 | sed "s|:|,|" > all.csv

3. remove characters from file names recursively:

find . -name '*\0*' | while read f; do mv "$f" "${f//\0/}"; done

4. remove last 3 characters from file names and retain the file extension

rename 's/(.*).{3}(\.md).*/$1$2/' *

5. remove leading 0's from file names recursively:

for FILE in `ls`; do mv $FILE `echo $FILE | sed -e 's:^0*::'`; done

6. Extract audio from video

ffmpeg -i input-video.avi -vn -acodec copy output-audio.aac

7. Extract audio from all videos in a directory

for f in *.mov; do ffmpeg -i "$f" -q:a 0 -map a "${f%.mov}.wav" ; done

8. Extract video clips from video

ffmpeg -ss 00:01:00 -i filename.mov -to 00:01:00 -c copy newname.mov

9. Move all files of a specified extension, within all subdirectories, to another directory

find test/ -name "*.csv" -exec mv {} test/ \;

10. Return the duration of all .mov files in a directory

for f in *.mov
do
  echo -n "$f "
  ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$f"
done

11. Split audio into multiple clips

• parselmouth
• pydub

11. Write all the file names in a directory to a .txt file

ls > contents.txt

12. Crop out black bars on the side of a video (batch)

for f in *.mp4; do
  ffmpeg -i "$f" -vf crop=1280:1024:380:38 "converted/$f"
done;

13. All Markdown in directory to HTML

for i in /some/directory/*.md; do pandoc -f markdown -t html -s "$i" > "$i".html; done;

14. Unzip all .gz files in directory:

gunzip -dk *.gz

15. Convert extensionless files into .txt files:

for f in * ; do
    mv "$f" "$f.txt"
done

16. Concatenante multiple .txt files to single .txt file:

cat * > 5gms_final.txt

17. Sum the second column of a grep return:

grep "individuals" all.txt | awk '{ SUM += $3} END { print SUM }'

18. Remove character from files in directory and save in place:

sed -e s/,//g -i *

19. File name uppercase letters to lower case:

for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done

20. Shell script (data_server.sh) for sshing into a server and starting a service

#!/usr/bin/expect -f
spawn ssh [email protected]
expect "password"
send "Shure*m78\r"
expect "$ "
send "screen -S tweet -d -m code-server —host 0.0.0.0\r"
expect "$ "
send "exit\r"
interact