How to...

[Matt]

How to preview first 10 lines of a gzipped file:

gunzip -c ./filename.gz | head -n 10

How to view the first 10 lines of a bam file header:

samtools view -h /sequence.bam | head -n 10

How to view the total size of a folder (or file):

du -sh /folder/path

How to list files with permissions and human readable file sizes:

ls -halt

How to filter bash history for a keyword:

history|grep keyword

How to erase everything in a file using vi/vim:

:1,$d

How to get paste to work properly with Vim:

:set paste

How to re-attach to a running docker containers and view logs in the console:

cd /folder/containing/docker-compose.yml/
docker-compose logs -f -t

How to clear a NodeBB plugin:

./nodebb reset -p nodebb-plugin-name

How to do an ldapsearch for a user:

ldapsearch -x -H ldap://your.ldap.database.domain.com:389 -D 'cn=admin,dc=client_domain,dc=com' -b 'dc=client_domain,dc=com' -w adminpassord "(cn=user_name_being_searched_for)"

How to detach from screen:

ctrl+a then ctrl+d

How to find tags from a remote git repository

git ls-remote --tags https://path/to/repository.git

How to calculate disk usage of current directory, retrieve only top 5 and ignore permission errors:

du -cBM --max-depth=1 ./ 2>/dev/null | sort -n | tail -n 5

How to calculate disk usage of system:

df -h

How to ssh tunnel through a jump server with an ssh key. In this case, key1.pem is required for access to the jump server. Access to the destination server does not require a key but can only be achieved through the jump server.

ssh -i /path/to/your/key1.pem -o "ProxyCommand ssh -W %h:%p -i /path/to/your/key1.pem username1@jump.server.com" username2@destination.server.com

How to use rsync to transfer files FROM the current directory on a local server TO a folder on a remote server through a jump server. In this case, key1.pem is required for access to the jump server. Access to the remote server does not require a key but can only be achieved through the jump server. Sudo access on the remote server requires a password.

rsync -avhR --delete --rsync-path="echo sudoPasswordOnRemoteServer  | sudo -Sv && sudo rsync" --partial-dir=./.rsync-partial -e "ssh -i /path/to/your/key1.pem" ./ \
username@remote.server.com:/destination_folder

How to use rsync to transfer files TO the current directory on a local server FROM multiple source folders on a remote server through a jump server. In this case, key1.pem is required for access to the jump server. Access to the remote server does not require a key but can only be achieved through the jump server. Sudo access on the remote server requires a password. A text file is also create that records files that cannot be accessed and excludes them from the rsync command.

exclude_file=./exclude.txt
rm ./exclude.txt

ssh -i /path/to/your/key1.pem -o "ProxyCommand ssh -W %h:%p -i /path/to/your/key1.pem username@remote.server.com" username@remote.server.com 'echo sudoPasswordOnRemoteServer | sudo -S find \
/source_folder_1 \
! -readable' >>"$exclude_file"

ssh -i /path/to/your/key1.pem -o "ProxyCommand ssh -W %h:%p -i /path/to/your/key1.pem username@remote.server.com" username@remote.server.com 'echo sudoPasswordOnRemoteServer | sudo -S find \
/source_folder_2 \
! -readable' >>"$exclude_file"

rsync -avhR --delete --rsync-path="echo sudoPasswordOnRemoteServer | sudo -Sv && sudo rsync" --exclude-from="$exclude_file" --partial-dir=./.rsync-partial -e "ssh -i /path/to/your/key1.pem" \
username@remote.server.com:/source_folder_1/ \
username@remote.server.com:/source_folder_2/ \
./

How to do a case-insensitve search in linux with the find command:

find / -iname "*stringYouAreLookingFor*"

How to exclude a character in a perl regular expression. Eg. to find anything other than #:

[^#]

How to specify the location of a perl library or module in the current directory. Place the folder containing the module in the same directory as the perl script (eg ./Algorithm/Loops.pm):

use lib '.' ;
use Algorithm::Loops 'NestedLoops';

How to identify tandem repeats in Perl:

my $dat = '01011010';
$dat =~ /(?=(.+)\1)(?!(??{ '.+?(..{' . length($^N) . ',})\1' }))/;
print "$1\n";

How to do indents in markdown:

          This text is indented.

How to stop markdown displaying links:

htt​ps://commons.wikimedia.org/w/index.php?curid=31927819

How to get the host address from inside a docker container:

ip route show default | awk '/default/ {print $3}'

How to hide all posts in a category (including from recent feed and unread feed) using css in NodeBB. This requires knowledge of the category number (82 in the case below):

li[data-cid ="82"] {
   display: none;
}

How to log in to non-running docker container:

docker run  -it image_name bash

How to inject Javascript code into ejs code:

<% code %>: Code is evaluated but not printed out
<%= code %>: Code is evaluated and printed out (escaped)
<%- code %>: Code is evaluated and printed out (not escaped)

How to export to path in Perl:

$ENV{'PATH'} .= ':'.'/new/path';

How to convert all bam files in a folder to sam:

for file in ./*.bam
do
    echo $file
    samtools  view -h $file > ${file/.bam/.sam}
done

How to extract a fastq read by it's title from a gz file:

gunzip -c 150901_I188_FCC7F64ANXX_L3_wHAMPI021865-80_1.fq.gz | grep -A 3 "HISEQ:378:C7F64ANXX:3:2109:10652:6636" >./found.1.fq

How to recursively delete a non-empty folder in Windows shell at the command prompt:

rd /S /Q "C:\path\to\folder"

How to find orphaned/abandoned uploads/files in NodeBB:
ACP->Manage->Uploads->Check for 'orphaned' status->delete

• NB. You have to go through all the files listed and find the ones that are labelled 'orphaned'

How to prevent empty line being inserted between divs in html:

<div style="padding:0;margin:0;clear:none;"></div>
<div style="padding:0;margin:0;clear:none;"></div>

How to check how much space docker container logs are using:

du -chs /var/lib/docker/containers/*/*json.log

How to store the output of a system call into a variable in Perl:

my $com='ls';
my $variable=qx($com 2>&1);

How to retrieve paths to all container docker volumes on the host:

docker inspect --format '{{range $mnt := .Mounts}} {{json $mnt.Source}} {{end}}' containerID

How to retrieve paths to one container docker volume on the host:

docker inspect --format '{{ range .Mounts }}{{ if eq .Destination "/data" }}{{ .Source }}{{ end }}{{ end }}' containerID

How to find out how much space docker container log files are using:

docker ps -qa | xargs docker inspect --format='{{.LogPath}}' | xargs sudo ls -hl

How to strip quotes surrounding a string in bash:

volumePath='"blah"';
volumePath=$(eval echo $volumePath);
echo $volumePath

How to use CSS to hide html tags with href that start with some text and end with some text:

a[href^="start_text"][href$="end_text"] {
    display: none; 
}

How to identify currency/money amounts/dollars with a regular expression in javascript:

^[0-9]+(\.[0-9]{1,2})?$

How to identify alphanumeric characters in a regular expression with javascript:

  this.match(/^[A-Za-z0-9]+$/);

How to remove all unused docker containers and volumes and networks:

sudo docker system prune -a
sudo docker network prune

How to convert text in a text box or textarea into uppercase in html:

text-transform: uppercase;

How to get a textarea in html to resize to the entered contents:

<script>
    $(document).on('input', 'textarea', function () {
        $(this).outerHeight(38).outerHeight(this.scrollHeight); // 38 or '1em' -min-height
    }); 
</script>

How to convert phred64 to phred33 in Perl and vice versa:

$newqual = join( "", map { chr( ord($_) - 64 + 33 ) } ( split( //, $oldqual ) ) );
$newqual = join( "", map { chr( ord($_) - 33 + 64 ) } ( split( //, $oldqual ) ) );

How to find your stripe client id:

https://dashboard.stripe.com/settings/applications

How to create an asynchronous code block in Node.js:

(async () => { ... })();

How to use await Promise.all to wait for an asynchronous array to finish in Node.js before proceeding:

/*This shows how to delete a list of files stored in a grid file system.*/
 
arrayOfFileNames=["file1.jpg", "file2.jpg", "file3.jpg"];
 

async function asyncArrayWait(arrayOfFileNames) {
    try {
        await Promise.all(
            arrayOfFileNames.map(async (fileName) => {
                console.log('file name  '+fileName);
                await gfsRemoveAsync(fileName);
            }),
        );

        function gfsRemoveAsync(param){
            return new Promise((resolve,reject) => {
                gfs.remove(param, function(err){
                    if (err) {
                        reject(err);
                    }
                    else{
                        resolve('done');
                    }
                });
            });
        }
        
        console.log('All files in the array have now been deleted');
    }
    catch (error) {
    console.log('There was an error:  '+error);         
    }
}

How to sequentially perform functions using await in Node.js:

let ID = '_idaghendhghner38393';

asyncRemoveStuff(ID);

async function asyncRemoveStuff(ID) {
    /* this function performs the three contained functions sequentially */
    try {

        /* This function depends on receiving the ID value from the parent function and returns its results in the obj2 variable*/

        let obj2= await gfsFindAsync({"metadata.ID":ID});
        function gfsFindAsync(param){
            return new Promise((resolve,reject) => {
                gfs.files.find(param).toArray(function (err, obj) {
                    if (err) {
                        reject(err);
                    }
                    else{
                        resolve(obj);
                    }
                });
            });
        }



        /* This function depends on receiving the obj2 value from above*/

        await gfsRemoveAsync(obj2);
        function gfsRemoveAsync(param){
            return new Promise((resolve,reject) => {
                gfs.remove(param, function(err){
                    if (err) {
                        reject(err);
                    }
                    else{
                        resolve('done');
                    }
                });
            });
        }


        /* This function depends on receiving the ID value from the parent function*/

        await mongooseRemove({ 'info.ID' :  ID });
        function mongooseRemove(param){
            return new Promise((resolve,reject) => {
                MongooseDB.remove(param, function(err) {          
                    if (err){
                        reject(err);  
                    else{
                        let msg='Removed from the database';
                        resolve(msg)                        
                    }
                });
            });
        }
    }
    catch (error) {
        console.log('There was an error:  '+error);        
    }
}

How to randomly generate a grid of pixels image with javascript and canvas and post it as a value in a form:

<html>
	<body>
		<canvas id="myCanvas" width="100" height="100" style="border:1px solid #d3d3d3;">Your browser does not support the HTML5 canvas tag.</canvas>

		<form action="/fund-setup" method="post" enctype="multipart/form-data">
			<input type="hidden" id="randomIcon" name="randomIcon" value="">
			<button type="submit" >Submit</button>
		</form>

		<script>
		  var canvas = document.getElementById('myCanvas'),ctx = canvas.getContext('2d'), x, y = 0, dim = canvas.width; ctx.translate(0.5, 0.5);                        
		  for(; y < dim; y=y+20) {                          
		      for(x = 0; x < dim; x=x+20) {
		          ctx.fillStyle = getRandomColor();          
		          ctx.fillRect(x, y, 20, 20);               
		      }
		  }
		  function getRandomColor() {
		      var r = 255*Math.random()|0,
		          g = 255*Math.random()|0,
		          b = 255*Math.random()|0;
		      return 'rgb(' + r + ',' + g + ',' + b + ')';
		  }
		  var canvasString = canvas.toDataURL();  
		  var randomIcon = document.getElementById('randomIcon');
		  randomIcon.value = canvas.toDataURL("image/jpeg", 0.5);        
		</script>

	</body>
</html>

How to convert a callback function into a promise and use it with async await. This code must be contained within an async code block:

let ID = '_idaghendhghner38393';

let obj2= await gfsFilesAsync({"metadata.ID":ID});

function gfsFilesAsync(param){
    return new Promise((resolve,reject) => {
        gfs.files.find(param).toArray(function (err, obj) {
            if (err) {
                reject(err);
            }
            else{
                resolve(obj);
            }
        });
    });
}

How to use promise.all in an asynchronous code block to wait until an array of async functions complete:

/*this is embedded in an async block*/
await Promise.all(
    fileNamesToDelete.map(async (ifile) => {
        /*the await below is important*/
        await gfsRemoveAsync({filename: ifile, root: 'images'});
        function gfsRemoveAsync(param){
            return new Promise((resolve,reject) => {
                gfs.remove(param, function (err) {
                    if (err) {
                        reject(err);
                    }
                    else{
                        resolve('done');
                    }
                });
            });
        }
    })
);

How to validate price fields with a regex:

/^(\d*([.,](?=\d{3}))?\d+)+((?!\2)[.,]\d\d)?$/

How to specify the location of modules in Perl and Raku:

#for the current directory
use lib '.' ;
#for the directory 'lib' in the current directory
use lib  'lib';

How to use functions from an external script in Node.js:

#in external.js:
module.exports = function() { 
    this.add = function(a,b) { return a+b };
    this.divide = function(a,b) { return a/b };
    //etc
}

#in main.js:
require('external.js')();
add(1,2);

How to center test in markdown:

<p style="text-align: center;">Centered text</p>

How to convert a number to x decimal places in javascript:

var cents=78.9
cents=cents.toFixed(2); //78.90
cents=78.997
cents=cents.toFixed(2); //79.00

How to launch the game of life with perl golf (no limits);

perl -e '$c=q(((x7a/(x13)X4Ax!18)1817@0)4);$,=$/;$/=AX3AAAx76;$b=pack("(A)*",map{rand 3<1}0..1816)x6;{print unpack"(a79)23a0",$b;select$v,$v,$v,0.1;$b=pack"(A)*",unpack"$c",pack"((a*)17xx!18)*",unpack"x1737(AA$/Ax$/AA$/@)2222",$b;$a++;redo}'

How to launch the game of life with perl golf (with limits);

perl -e '$a=0;$c=q(((x7a/(x13)X4Ax!18)1817@0)4);$,=$/;$/=AX3AAAx76;$b=pack("(A)*",map{rand 3<1}0..1816)x6;{print unpack"(a79)23a0",$b;select$v,$v,$v,0.1;$b=pack"(A)*",unpack"$c",pack"((a*)17xx!18)*",unpack"x1737(AA$/Ax$/AA$/@)2222",$b;$a++;redo if $a <50;}'

How to control the width of a text box and make sure it is in the center of the page using inline CSS and html:

<input type="text" style="max-width: 350px; margin:auto;"  id="inputName" name="inputName">

How to dump a database from a non-standard port with mongodb:

mongodump --port 87017 -d databaseName -o ./dump/databaseName-20200503

How to migrate a nodebb mongodb to a new server:

• Backup the nodebb database on the old server:

mongodump -u nodebb -p [password] -d nodebb -o dump/nodebb-2018-03-30

• On the new server, restore the nodebb database from the backup:

mongorestore -u nodebb -p [password] -d nodebb dump/nodebb-2018-03-30/nodebb/

• NB. It seems you have to reinstall/upgrade NodeBB after this, otherwise there are strange errors with pages not loading properly unless you reload them etc.

How to find IP addresses of users who have successfully logged in to an Ubuntu/Debian server:

zgrep sshd /var/log/auth.log* -h |grep -F 'Accepted'

How to recreate a Dockerfile from a docker image:

#NB. unfortunatlely if an ADD or COPY command was used you cannot work out what files were added/copied
docker history --no-trunc=true image1 > image1-dockerfile

How to get the IP address of a Docker container:

docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $INSTANCE_ID

How to get the IP address of a running Docker container that has a hyphen in it's name and save it in a variable:

IP=$(docker inspect --format '{{$network := index .NetworkSettings.Networks "container-name_default"}}{{$network.IPAddress}}' $(docker ps -q))
echo $IP

How to find an element in an array within an object and only display the desired element in MongoDB (the $ is important!):

db.things.find({"stuff.arraystuff": {"$elemMatch": {"ID": '1b6c0f70dd',"nr":'908484f1127c'}}}, {"stuff.arraystuff.$": 1})

How to reset/re-activate cut and paste function to/from VirtualBox linux guest and Windows host:

#in linux guest:
killall VBoxClient
VBoxClient-all

How to draw on top of a pdf:

Tools->Comment->[select pencil-drawing tool-icon]

How to load an R data file (.RData) into a variable in R:

var1 <-  get(load("D:/path/to/file.RData"))

How to encode dummy variables in R using the Caret package:

dummy_variablez <- dummyVars(" ~ .", data = variablez)

How to manually encode a dummy variable in R:

df$female <- ifelse(df$sex == female', 1, 0)

How to convert a range of columns in R to numeric (from character):

#without dplyr:
dataz[,2:6] <- sapply(dataz[,2:6],as.numeric)

#using dplyr:
if (!require(dplyr)) install.packages("dplyr")
library(dplyr)
library(dplyr)
dataz[,2:6] %>% mutate_if(is.character,as.numeric)

How to change the label of a checkbox with JQuery:

<label id="lblfilter-green" class="btn btn-outline-secondary btn-sm"><input type="checkbox" name="direction" id="filter-green" /><span>&nbsp;descending</span></label>
<script>
    $('label').click(function () {
        var checked = $('input', this).is(':checked');
        $('span', this).text(checked ? ' ascending' : ' descending');
    });
</script>

How to log into a screen session from multiple computers:

screen -x nameofsession

How to preview an image after selecting it in html:

<img id="blah" width="auto" height="100" />
<input  type="file" name="myTopImage" onchange="document.getElementById('blah').src = window.URL.createObjectURL(this.files[0])" />

How to submit a form when you select an image in html:

<input  type="file" name="myImage" onchange="form.submit()" />

How to install local R packages in RStudio:

if (!require(ThePackage)) install.packages("/path/to/the/package.tar.gz",repos = NULL, type="source")

How to remove the "Powered by" footer in NodeBB:

ACP > Extend > Widgets, look at Global Footer.

How to process all files of a specific type in a specific directory using bash:

	dir="."
	file="*.list"
        #the if statement below prevents any error message if the file type doesn't exist
        if ls ${dir}/$file >/dev/null  2>&1; then
	        for file in `cd ${dir};ls -1 ${file}` ;do
	           echo $file
	        done  
        fi

How to add elements to a list using a for loop in R:

listz <-rep(0,8)
for (i in 1:length(listz)){
  print(i)
  listz[[i]] <- i

}
print(listz)

How to sort an object by keys in javascript:

        //object is obj
	Object.keys(obj).sort()
      	.forEach(function(v, i) {
        	console.log(v, obj[v]);
       	});

How to calculate a floating point percentage (with eight digits) in bash from variables with bc and assign it's value to a variable in bash:

PCT=`echo "scale=8; ($NO / $TOTNO)*100" | bc`

How to read all lines in a file including the last line without a newline character in bash:

input=file.txt
while read line || [ -n "$line" ]
do
   echo $line
done < $input

How to match fq, fastq,fq.gz,fastq.gz files with a regex in bash:

fileFormat="$1"

if [[ $fileFormat =~ [Ff]([Aa][Ss][Tt])?[Qq](\.[[:alnum:]]+)?$ ]]; then
    echo "its a FASTQ file"
else
    echo "its not a FASTQ file"
fi

How to calculate the length of multi-line fasta records in multi-fasta files using bash / awk:

awk '/^>/{if (l!="") print l; print; l=0; next}{l+=length($0)}END{print l}' ./sequence.fasta

How to get the nextflow_dir path of the previous container in Nextflow and delete a file in it:

#in the previous container set the file in an output command:
output:
     set file("${pair_id}_raw_snps_${round}.vcf") into raw_snps_ch

#in the next container set a val with an input command:
input:
     val(raw_snps) from raw_snps_ch

#In the script section, the $raw_snps variable will have the full path of the file in the nextflow_dir folder. You can then delete it:
script:
'''     
rm $raw_snps
'''

How to count the number of folders in a path in bash (i.e the number of levels in a directory structure) and store it in a variable:

varz=`echo "/directory/structure/" | tr " /" "- " | wc -w`

How to untar a tar.gz file and omit the preceding directory structure:

#first count the number of levels in the preceding directory structure to be omitted
varz=`echo "/preceding/directory/structure/to be omitted/" | tr " /" "- " | wc -w`
#in this case we are extracting it into a directory that is the same as the preceding directory to be omitted
#in addition we are using the k option so that extracting does not overwrite existing files/folders
tar --strip-components $varz -C /preceding/directory/structure/to be omitted/ -zxvkf /tarfile/tobeextracted.tar.gz

How to sort all columns in a pandas dataframe and reset the index in Python:

   df.sort_values(by=df.columns.to_list(), axis=0, ascending=True, inplace=True, kind='quicksort', na_position='last')
   df=df.reset_index(drop=True)

How to transpose a 2D array in Python:

2d=[
['A', 'B', 'C', 'D'],
['E', 'F', 'G', 'H'],
['I', 'J', 'K', 'L']
]

2dStar=list(zip(*2d))
print(2dStar)

How to create multiple columns at once in a dataframe with Python and Pandas using series:

'''
Starting DF:
   A  B
0  6  1
1  8  4
Desired DF:
   A  B  C   D
0  6  1  16  56
1  8  4  18  58
'''
def myfunc1(row):
    C = row['A'] + 10
    D = row['A'] + 50
    return pd.Series([C, D])

df[['C', 'D']] = df.apply(myfunc1 ,axis=1)

How to apply a function over multiple columns in a Python Pandas dataframe to create a new column:

import pandas as pd

df = pd.DataFrame({'ID':['1', '2', '3'], 'col_1': [0, 2, 3], 'col_2':[1, 4, 5]})
mylist = ['a', 'b', 'c', 'd', 'e', 'f']

def get_sublist(sta,end):
    return mylist[sta:end+1]

df['col_3'] = df.apply(lambda x: get_sublist(x['col_1'], x['col_2']), axis=1)

'''
Output:
  ID  col_1  col_2      col_3
0  1      0      1     [a, b]
1  2      2      4  [c, d, e]
2  3      3      5  [d, e, f]
'''

How to do matrix multiplication with a for loop or the numpy dot function (faster) in Python:

   import numpy as np

    np.random.seed(seed)
    A = np.random.rand(n,n)
    B = np.random.rand(n,n)
    C = np.zeros((n,n))

    #With for loop:

    for i in range(n):
        for j in range(n):
            for k in range(n):
                C[i,j] += A[i,k]*B[k,j]

    #With dot function:

    C += A.dot(B)
                

How to initialise values in a dictionary in Python with defaultdict:

from collections import defaultdict

D2 = defaultdict(int) 

#note below you don't first have to initialise the value to 0
D2['new-key'] += 1

print(D2)

How to iterate a list to get the index and value using the enumerate function in Python:

for indexz,valuez in enumerate(listz):
    print(indexz,valuez)

How to split a string up into individual characters in Python:

#the list function does this if given a string
stringz='atcgcgta'
listz=list(stringz)

How to cut a string at a range of positions provided as a list in Python:

def get_slices(s, l):
    l = [0] + l + [len(s)]
    cut =[s[x:y] for x,y in zip(l, l[1:])]
    return cut

s='GCATAGTAATGTATTAATGGC'
l=[7, 15]
sliced=get_slices(s,l)

How to return the indexes of values in a list that pass a condition in Python:

lz=[6, 9, 10, 6, 12]
newList = [ i for i,t in enumerate(lz) if t  >= 9]

How to check if an element occurs in a list in Python:

ex='slot'
listz=['luck', 'slot', 'just', 'opened', 'kitchen']
if ex in listz:
    print ("found in list ",ex)
if ex not in listz:
    print (ex," is not in the list ")

How to retrieve key:value pairs from a dictionary that has keys common with another dictionary in Python:

d1={'a':1,'c':1}
d2={'a':1,'b':1,'c':1}
common_dict = {x:d1[x] for x in d1 if x in d2} 

How to retrieve keys common to two dictionaries in Python 3:

x={'a':1,'c':1}
y={'a':1,'b':1,'c':1}
for key in x:
    if key in y:
        print (key, x[key])

How to sort a dictionary in descending order by it's values and retrieve the first k pairs in Python:

sorted_dictz=sorted(dictz.items(), key=lambda x: x[1], reverse=True)[:k]

How to get a list of indices of x elements with the highest values from a list in Python:

x=5
listz=[1,6,3,9,6,7,6,8,3,5]
top_5_element_indexes = sorted(range(len(listz)), key = lambda sub: listz[sub])[-x:]

How to sum all the values in a 'value column' of a pandas dataframe according to strings in a 'key column' and then use those summed total values to create a new column that contains the value/total_value fraction for each occurence of the key in the original 'key column' :

tot_value=df.groupby('key')['value'].sum()
df['fraction']=df.apply(lambda x: x['value']/tot_value[x['key']], axis=1)

How to compare two Pandas dataframes and retrieve the differences:

#you can also have sort=False
pd.concat([df1, df2],sort=True).drop_duplicates(keep=False)

How to retrieve the first n elements of a set:

for n,elem in enumerate(set_name):
    if n == 5:
        break
    print (elem)

How to simultaneously iterate through multiple lists of the same length in Python:

    for r,c,v in zip(list1,list2,list3):
        print("r:",r)
        print("c:",c)
        print("v:",v)

How to sort and find unique elements in a list in Python (2 methods):

    method1=sorted(set(listz))
    method2=sorted([*{*listz}])

How to do element-wise multiplication of two lists to create a new list in Python:

newlist=[i * j for i, j in zip(list1, list2)]

How to merge two Pandas dataframes using columns with different names in Python:

df3=pd.merge(df1,df2[['Code','Desc']],left_on='ID',right_on='Code',how='inner')

How to generate a random matrix of defined shape with floating point numbers between a minimum and maximum value in Python with numpy:

import numpy as np
m,n=(3,4)
min,max=-2.5,3.4
rand_matrix=np.random.uniform(min,max,(m*n)).reshape(m,n)

How to concatenate Python pandas dataframes with different column names and remove indexes:

import numpy as np
#this trick uses numpy to remove the column names and indexes
df = pd.DataFrame( np.concatenate( (df1.values, df2.values), axis=0 ) )
#add column names in as a final step
df.columns = [ 'a', 'x', 'y' ]

How to retrieve multiple columns of a pandas dataframe by their name using regex:

dfSub=df.filter(regex = 'City|Country|Region')

How to convert dates to timestamps (down to sub millisecond resolution) in a Python Pandas dataframe:

df['Timestamp']=pd.to_datetime(df['Date'],errors='coerce')
df['Timestamp'] = df['Timestamp'].dt.strftime('%Y-%m-%d %H:%M:%S:%f')

How to concatenate multiple pandas dataframe columns in python:

newdf =pd.concat([df['col1'], df['col2'],df2['col1'],df2['col2']], axis=1, ignore_index=True)

How to filter Python pandas dataframe by multiple conditions:

is_1 = (df["Col1"] == "hello")
is_2 = (df["Col2"] == "is_it_me")
df_fil=df[is_1 & is_2]

How to do left, right, inner and outer joins in Python with pandas dataframes:

#outer merge:
result = pd.merge(user_usage,
                 user_device[['use_id', 'platform', 'device']],
                 on='use_id')
#left merge:
result = pd.merge(user_usage,
                 user_device[['use_id', 'platform', 'device']],
                 on='use_id', 
                 how='left')

How to rename a column in a Python pandas dataframe:

data.rename(columns={'gdp':'log(gdp)'}, inplace=True)

How to remove / drop multiple columns in a Python pandas dataframe:

result.drop(['Name1','Name2'], axis=1, inplace=True)

#OR:
del df_states["Name1"]
del df_states["Name2"]

How to copy a Python pandas dataframe:

df2=df.copy(deep=True)

How to extract only column 1 and 2 from a Python pandas dataframe and group by values in column 1 and then sum the aggregated column 2 values into a column named 'count' and then sort them in descending order:

    sum_sorted_grouped_df = df[['Col1','Col2']].groupby(['Col1'])['Col2'] \
                         .sum() \
                         .reset_index(name='count') \
                         .sort_values(['count'], ascending=False)

How to sort a Python pandas dataframe by values in two separate columns (the first ascending and the second descending):

df.sort_values(['b', 'c'], ascending=[True, False], inplace=True)

How to change the dtype of a Python pandas dataframe column:

df['Colz']=df['Colz'].astype('float64')

How to create a dictionary from two columns in a Python pandas dataframe:

dictz=dict(zip(df['col1'],df['col2']))

How to create an empty pandas dataframe with column names in Python:

column_names = ["a", "b", "c"]
df = pd.DataFrame(columns = column_names)

How to create a dictionary from an anonymous list in Python:

a=dict([['w',1],['y',2]])
print(a)
#{'w': 1, 'y': 2}

How to generate consecutive pairs from a list in Python:

l = [1, 7, 3, 5]
for first, second in zip(l, l[1:]):
    print (first, second)

#1 7
#7 3
#3 5

How to generate pairs from two columns in a Python pandas dataframe and assign them as values using keys in another column:

#generate pairs from two columns:
 zipz=zip(df['col2'],df['col3'])
#create dictionary with keys in another column
 dictz=dict( zip( df['col1'],zipz))
#output:
#{0: (999, 556), 1: (153, 995), 2: (679, 075)}

How to generate a COO matrix of m by n dimensions in Python with Scipy:

R = [0, 0, 1, 1, 2, 2]
C = [1, 2, 0, 1, 0, 1]
V = [5, 2, 1, 1, 6, 1]
import scipy.sparse as sp
coo_matrix((V, (R, C)), shape=(n, m))

How to do COO and CSR matrix multiplication with Python Scipy:

import numpy as np
import scipy.sparse as sp

R = [0, 0, 1, 1, 2, 2]
C = [1, 2, 0, 1, 0, 1]
V = [5, 2, 1, 1, 6, 1]
P=  [0,    2,    4,   6]

x = [1, 2, 3]

x = np.ones(num_verts)

#COO
COO = sp.coo_matrix((V, (R, C)))
COO.dot(x)

#CSR
CSR = A_coo_sp.tocsr() # or...    sp.csr_matrix((V,C,P))
CSR.dot(x)

How to write a function and apply it to a Python numpy matrix using vectorize:

    Y = np.array([[5, 2, 0],
                  [2, 4, 7]])
    print(Y)
    #[[5 2 0]
    # [2 4 7]]
    def myfunc(a):
        #Return 1 if a>0, otherwise return 0
        if a > 0:
            return 1
        else:
            return 0
    vfunc = np.vectorize(myfunc)
    
    A=vfunc(Y)
    print(A)

    #[[1 1 0]
    # [1 1 1]] 

How to remove duplicated rows in a pandas dataframe in python:

df.drop_duplicates(subset = None, inplace=True)

How get the index positions of elements in a numpy array that satisfy a filter condition using the 'where' function in Python and use them to retrieve their corresponding values:

x = np.array([-4, 4, 5])
from numpy import where
k = where(x > 0)[0]
#k=[1 2]
y=x[k]
#y=[4,5]

How to create an empty array in numpy using the empty() function in Python:

#The values in the array are whatever is located in that memory address at the time.
#This is quicker than creating an array of zeros
from numpy import empty, arange
kx = empty(10, dtype=int)
print(kx)

How to use a list of index positions to reorder values in a Python numpy array:

indexes=[4,3,2,1,0]
values=[10,9,8,7,6]
new=np.array([0] * len(values))
new[indexes]=values
print (new)
#[ 6  7  8  9 10]

How to get a list of index positions (array3) of elements in an array (array 1) that match elements in another array (array2) using numpy and Python:

import numpy as np
array1=np.array([0,3,5,6,9,1,2,4,7,8])
#empty, dtype=int creates a list of length n. It has values equal to whatever was at that memory address at the time)
#as an alternative to empty() you could use:
#array3=np.array([0] * len(array1))
#however it is slower
array3 = empty(len(array1), dtype=int)
array2=np.array([0,1,2,3,4,5,6,7,8,9])
#The line below is where the magic happens. TBH I'm not so sure why it works, however it is not the same as array3=array1[array2]
array3[array1] = array2
print(array3)
#[0 5 6 1 7 2 3 8 9 4]

How to count the occurences of a string in a column of a Python pandas dataframe:

counts = df['TITLE'].value_counts()

How to randomly pick a sample of fixed size from a numpy array in Python (without replacement):

import numpy as np
population=np.array([0,1,2,3,4,5,6,7,8,9])
samplez = choice(population, size=5, replace=False)

How to reverse an array in Python using a subscript with two colons followed by minus 1:

#b is the reverse of a
b=a[::-1]

How to get a list of indexes of an array in the order they would be in if the array values were sorted:

x = np.array([3, 1, 2])
np.argsort(x)
#[1, 2, 0]

How to concatenate strings in two separate columns of a pandas dataframe in Python:

df['concat']=df['col1']+'_'+df['col2']

How to calculate the mean of numpy arrays while ignoring any nan values in Python:

arrayz= np.array([[np.nan,  21.85, np.nan], [  6.95, np.nan, -31.55]])
meanz=np.nanmean(arrayz)

How to create a comment block in a bash script:

: <<'END'
comments go here
END

How to clear the contents of a text file in Python:

open('output.txt', 'w').close()

How to remove a file extension and store it in a variable in bash:

filename='file.txt'
base=${filename%.*}
echo $base
#file

How to convert a Nextflow array into a bash array within a NextFlow shell script block:

#reads='[1.fastq.gz, 2.fastq.gz]'
#remove square brackets
nfarray=\$(echo "${reads//[][,!]}")
#split at commas
arr=(`echo \$nfarray | cut -d ","  --output-delimiter=" " -f 1-`)
#print out first element
echo "\${arr[0]}"


#######################
#NB. When I retested the above code I seemed to get an error. Upon further testing it seemed that you can directly use nextflow arrays thus:

for filename in ${reads}; do
	filename=\$(echo "\${filename//[][,!]}")
	echo \$(basename \${filename})
done

How to filter vcf files with bcftools:

#http://samtools.github.io/bcftools/bcftools.html#expressions
bcftools filter --regions-file $pathToBed/BEDFILE.bed $inputVCF | bcftools filter -e "QUAL < 10 || F_MISSING > 0.01 || MAF > 0.98" | bcftools view -m2 -M2 -v snps --output-type z --output-file $outputDIR/$name._filtered.vcf.gz | bcftools index -t --output-file $outputDIR/$name._filtered.vcf.gz.tbi

How to use eval in Perl to execute text as code:

$name='yes';
$reg='s/yes/no/g';
my $code='$name'.' =~ '.$reg.';';
eval($code);
print "$name\n";
#no

How to use bash to find the longest common substring between two strings:

				word1=$b1
				word2=$b2
				if [ ${#word1} -lt ${#word2} ]
				then
					word1=$b2
					word2=$b1
				fi
				comsub=''
				for ((i=${#word2}; i>0; i--)); do
					for ((j=0; j<=${#word2}-i; j++)); do
						if [[ $word1 =~ ${word2:j:i} ]]
						then
							echo ${word2:j:i}
							comsub=${word2:j:i}
							break 2
						fi
					done
				
				done

How to only move a file if it exists in bash:

[ ! -f a.txt ] || mv a.txt b.txt

How to use eval to process regular expressions supplied as text strings in Perl:

my $string='cat';
my $reg='s/a/un/';
my $code='$string'.' =~ '.$reg.';';
eval($code);

How to use grep to find a regex in a directory of gzipped files in linux:

find /path/to/dire -name \*.gz -print0 | xargs -0 zgrep -n -E 'GCGT|N|Y|y|W|w|K|k|V|v|U|u'

How to use bash with paste and shuf to randomly shuffle paired fastq files:

paste <(zcat test.2.fq.gz) <(zcat test.2.fq.gz) | paste - - - - | shuf | awk -F'\t' '{OFS="\n"; print $1,$3,$5,$7 > "random.1.fq"; print $2,$4,$6,$8 > "random.2.fq"}'

How to extract specific lines from an archived gzipped gz file (and print out the line numbers):

gunzip -c file.gz | awk -v from=10 -v to=20 'NR>=from { print NR,$0; if (NR>=to) exit 1}'

How to strip leading and trailing quotes from a string in Python:

    if (string[0] == string[-1]) and string.startswith(("'", '"')):
        string=string[1:-1]

How to get the path, file name, base name and extension in bash:

full_name="/foo/fuzz.bar"
file_name=$(basename ${full_name})
base_name=${file_name%%.*}
extension="${filename##*.}"
path=${full_name##*/}

How to process samples with multiple paired reads per sample using NextFlow:

https://stackoverflow.com/questions/69702077/nextflow-how-to-process-multiple-samples

How to change a NextFlow/Groovy channel variable in a script and then put it back in a channel (file method):

	process changer {
		cpus 1
		memory '1 GB'
		time '1m'
	   input:
			val(variable1) from ch1
			
		output:
			file(*) into ch2
		script:

			"""
			echo "variable1 is: "$variable1
			variable2="hello: $variable1"
			touch \$variable2
			"""
	}

	process reintroducer {
		cpus 1
		memory '1 GB'
		time '1m'
		
	        input:
			//How to get the base filename without extensions in Nextflow with simpleName
			val(variable3) from ch2.map { file -> file.simpleName }		
		output:
			val (variable3) into ch3
		script:
			"""
			echo "variable3 is:"$variable3
			"""
	}

How to apply a regex to a channel variable in NextFlow:

	process getSimpleName {
		//scratch true
		cpus 4
		memory { 0.GB +(24.GB * task.attempt) }	
		time { 14.hour + (8.hour * task.attempt) }
		errorStrategy { (task.exitStatus in 1..272 && task.attempt <= maxRetries)  ? 'retry' : 'terminate' }
		maxRetries 5	


		input:
			val(modified_names) from original_names_ch.map ({ varz -> varz.replaceFirst(/\*\//,"").replaceAll("\\.txt", "") })


		output:			
			val(modified_names) into modified_names_ch


		script:
			"""
                        echo "stripped proceeding paths and txt file extension to create "$modified_names
			"""
	}

How to write a regex to identify all word characters (including _) in a variable in bash:

string='abcd123'
regex='([_[:alnum:]])'
				
if [[ "$string" =~ $regex ]]; then
    echo "word characters found"
fi					

How to test for the existence of unzipped or gzipped fasta files using a Perl one-liner:

		flag=`perl -e 'BEGIN {$t=shift};my $fz=0;  if($t =~ /\.(fa|fasta|fa\.gz|fasta\.gz)$/){$fz=1};print "$fz"' ${file} `
			
		echo "flag is: "$flag

How to alphanumerically sort files that end with a number and store the results in a bash array using a Perl one-liner:

#NB print0 appends null to the end. Consequently .$ must be present in the statement map {/(\d+).$/; [$_, $1];}
array3=("`find ${input} -type d  -mindepth 2  -print0 | perl -e '$/="\0"; my @files=<>; my @sorted_array = map { $_->[0] } sort { $a->[1] <=> $b->[1]} map {/(\d+).$/; [$_, $1];} @files; print "@sorted_array";'`")

How to get file attributes such as basename and filesize in NextFlow:
https://www.nextflow.io/docs/latest/script.html#check-file-attributes

How to collect all of the files produced by a NextFlow process and analyse each of them individually in separate processes:

	process createFiles{
		cache false
		cpus 2
		memory '4 GB'
		time '4h'

		input:
			val(refz) from ref_ch2
			set val(pair_id), val(input_bam) from bam_for_intervals.mix(bam_for_intervals2).mix(bam_for_intervals3)	

		output:
		
			path ("./intervals/*.list") into intz_ch
			val (input_bam) into bam_for_variant_calling
			
		publishDir "${params.intervals}", mode:'move'
		
		script:
			scriptdir=params.scripts
			intervaldir=params.intervals

			"""
			filename=$input_bam
			filename=\$(echo "\${filename//[][,!]}")		
			b1=\$(basename \$filename)
			input_bamz=${params.bam}/\${b1}
			
			dir=$params.ref
			filed="*.fa"
			for file in `cd \${dir};ls -1 \${filed}` ;do
			   rf=$params.ref/\$file
			done			
			base=\${rf%.*}
			refdict="\${base}.dict"	
			refname=\$(basename \$file)
			perl $scriptdir/fastaq2bed/fastaq2bed.plx  -i \$rf -o \$rf   || true
			java -jar \$PICARD_JAR BedToIntervalList \
				  I=\${rf}.bed \
				  O=\${refname}.interval_list \
				  SD=\${refdict}	
			perl $scriptdir/perl_mutect2/perl_mutect2.plx  -q \${rf} -i \${input_bamz} -s \${refname}.interval_list -x 'intervals'  -k ${params.max_interval_bp} -o ./   || true	  
			#dir="./intervals"
			#filed="*.list"
			#for file in `cd \${dir};ls -1 \${filed}` ;do
			   #mv \${dir}/\${file} "${params.intervals}/"
			#done 
				  
			"""
	}

process indexer {
	//NB. pgen_ch2 does not increment if cache is not set to false and is therefore essential
	//cache false
	//scratch true

	cpus 1
	memory '2 GB'
	time '1h'

    input:
		each (input_interval) from intz_ch.collect()
    output:	
		file ("*.list") into pgen_ch3
    script:
	
    """
		
		echo "input_interval is ${input_interval}"
		file_name=\$(basename ${input_interval})
		echo "file_name is \${file_name}"
		touch \${file_name}
    """
}

process haplotypeCaller {
	scratch true
	cpus 4
	memory { 0.GB +(24.GB * task.attempt) }	
    time { 14.hour + (8.hour * task.attempt) }
    errorStrategy { (task.exitStatus in 1..272 && task.attempt <= maxRetries)  ? 'retry' : 'terminate' }
    maxRetries 5	

    input:
		set val(refz), val(bam_name), val(input_interval) from ref_ch3.combine(bam_for_variant_calling.combine(pgen_ch3))

    script:
		round=1
		"""
		
		echo "input_interval is "${input_interval}
		echo "bam_name is "${bam_name}
		ref_name=\$(basename ${refz})
		ref_base=\${ref_name%%.*}
		ref=\${ref_base}		
		
		file_name=\$(basename ${bam_name})
		base_name=\${file_name%%.*}
		pair_id=\${base_name}
		
		echo "pair_id is " \${pair_id}
		inz="${input_interval}"
		interval_base_name=\${inz%%.*}
		interval_num="\$interval_base_name"
		regex='([0-9]+).list'
		[[ \$interval_num =~ \$regex ]]
		echo \${BASH_REMATCH[1]}
		
		
		if [ "$params.doJointCallingBQSR" == "true" ] || [ "$params.doJointCalling" == "true" ]; then
			echo "Doing joint calling"
			gatk HaplotypeCaller \
			-R \$ref \
			-I ${params.bam}/$bam_name \
			-L ${params.intervals}/\$interval_base_name \
			--emit-ref-confidence GVCF \
			-O ${params.vcf}/\${pair_id}_raw_variants_${round}_\${BASH_REMATCH[1]}.g.vcf
		else	
			echo "Not doing joint calling"
			gatk HaplotypeCaller \
			-R \$ref \
			-I ${params.bam}/$bam_name \
			-L ${params.intervals}/\$interval_base_name \
			-O ${params.vcf}/\${pair_id}_raw_variants_${round}_\${BASH_REMATCH[1]}.vcf 
		fi
		"""
}

How to calculate md5 hash for a file in windows:

certutil -hashfile file.txt MD5

How to calculate the md5 checksum for all files of a certain type in a folder using the Windows command line cmd.exe:

#set the target folder, file type and output folder and paste the code below into the cmd shell.
#md5 checksums are written to \output_folder\md5.txt for each file and then sorted and an md5 checksum is created from the sorted text file
#The final results are in \output_folder\md5_final.txt
set "target_folder=E:\test"
set "file_type=*.txt"
set "output_folder=E:"
type NUL > %output_folder%\md5.txt
for /R %target_folder% %f in (%file_type%) do @certutil -hashfile "%f" MD5  | findstr /V ":" >>%output_folder%\md5.txt
sort /M 10240 %output_folder%\md5.txt > %output_folder%\md5_sorted.txt & certutil -hashfile %output_folder%\md5_sorted.txt MD5 > %output_folder%\md5_final.txt

How to search for a file and retrieve it's path in linux:

find / -name 'libperl.so'  -exec readlink -f {} \;

How to print @INC with a perl one-liner:

perl -e 'print "@INC"'

How to check for a matching file extension in bash:

echo "file name is read1.gz"
if [ "${read1##*.}" = "gz" ]; then
    echo "this is a gz file"
fi

How to check if a specific file type / file extension exists in a directory in bash:

dir="/path/to/dir"
file="*.txt"
if ls ${dir}/$file >/dev/null  2>&1; then
   echo $file" exists"
fi

How to set the contents of a channel into two separate channels in Nextflow:

Channel.fromPath("${params.path}/*.*cf").into({channel_1;channel2})

How to fix the error

[0m/.singularity.d/actions/exec: exec: line 9: /bin/bash: not found

with NextFlow and singularity containers built on alpine linux:

• Install bash when building the container

RUN apk add bash

How to make a NextFlow file channel send an empty file with a specific filename ("empty_file") if the channel is empty:

Channel.fromPath( "/home/*.{txt,pdf}" ).ifEmpty(file("empty_file")).set { new_ch }

How to remove the file extension from a filename generated by a path channel in NextFlow:

val(ext) from ch.map({path -> path.getFileName().normalize().toString().split(/\./)[0]})

#if you get the error "Unknown method invocation `getName` on ArrayList type". Try the following:

val(ext) from ch.map({path -> path[-1].getFileName().normalize().toString().split(/\./)[0]})

How to remove the preceding file path and get the base name from a filename generated by a path channel in NextFlow:

val(basename) from ch.map({path -> path.getFileName().normalize().toString().split(/\//)[-1]})

How to separate each element of an array in a file channel into individual channels in Nextflow. (This example also converts file paths to base names):

#flatten may not always be required
each (val) from ch.map ({ file -> file.simpleName }).flatten()

How to modify just one element of an array in a channel in NextFlow:

#for the output channel of the previous process:
set val(val1), val(val2), file ("*.txt") into output_ch


#for the input channel of the next process (this modifies the third element to get the file name of a file). Note how square brackets must be put around the output if you don't want the input flattened:

set val(valz1), val(valz2), val(filez) from output_ch.map ({ val1, val2, file -> [pair_id, round, file.getName()] })

How to copy files from a docker container using a wildcard:

#the secret is to use a . instead of *
docker cp container_id:/container_folder/. ./host_folder/

How to do non-integer maths in bash with bc and get an answer with a leading 0 in front of the decimal point and 6 decimal places and save it into a bash variable:

rd1="scale=6; ${P}/${tot_reads}"
rd2=`printf '%.6f' "$(echo $rd1 | bc)"`

How to increment a counter in Bash:

COUNT=$((COUNT+1))

How to create an array in Bash by splitting a comma delimited string and then increment each value in the string by accessing each array element by it's index:

UM='1,2,3,4,5,6'
UPM=($(echo $UM | tr "," "\n"))

AM=''
for i in ${!UPM[*]}; do 
	echo ${UPM[$i]}
	ADDZ=`echo ${UPM[$i]}+1 | bc`
	AM=${AM}','${ADDZ}
done
	
UMF=`echo $AM | sed 's/^,//'`

How to delete all text in Vi text editor:

:1,$d.

How to enter insert mode in Vi editor:

press 'i'

How to exit insert mode in Vi editor:

press 'esc'

How to save and exit in Vi editor:

:wq! and enter

How to exit without saving in Vi editor:

:q! and enter

How to pipe to and from bcftools using STDIN and STDOUT (standard input and standard output):

#Ou is critical for STDOUT and - is critical for STDIN
bcftools norm -d none ./infile.vcf.gz -Ou | bcftools sort - –T ./temp -Oz -o ./outfile.vcf.gz

How to use brace expansion in bash to use ls to list all files that match multiple different patterns:

pattern1='*.zip'
pattern2='*.gz'
dir='/bin'
filez=`echo {${pattern1},${pattern2}}`
#you could also use: filez=`echo {*.zip,*.gz}`
echo "filez is: "$filez
for file in `cd ${dir};ls ${filez} 2>/dev/null` ;do
	echo \$file
done
				

How to replace non-printable characters in Perl (2 methods):

#Method 1:
$v =~ s/\P{IsPrint}//g;
#Method 2:
$v =~ tr[\0-\x08\x0B\x0C\x0E-\x1F\x7F][\x{2400}-\x{2408}\x{240B}\x{240C}\x{240E}-\x{241F}\x{2421}];

How to use an IF statement in a Nextflow process to choose between two different input channels using a ternary operator:

#uses ch1 if params.use_ch1 is true
val(A) from (params.use_ch1 == "true" ? ch1 : ch2)

How to convert a file containing channel generated by a NextFlow process to a value containing channel:

process generate{
	input:
		val (in) from in_ch
	output:
		file("*.{txt,doc}") into chz
	script:
		"""
		echo "$in"
		touch a.txt
		touch b.doc
		"""
}
//The channel ch1 now contains the filename surrounded by []
ch1=chz.collect ({ it.getName() })

process analyse{
	input:
		//flatten is required to remove the surrounding [] (and to launch one process per file name)
		val (valz) from ch1.flatten()
	script:
		"""
		echo "$valz"
		"""
}

How to do division in bash and give an answer as a whole number rounded up and store it in a variable:

mem=24
cpuz=9
memz=`echo "scale=0; (($mem + 0.5)/ $cpuz)" | bc`

How to combine multiple Nextflow channels together into one channel:

input:
	val(mixed) from ch1.collect().mix(ch2.collect()).mix(ch3.collect()).mix(ch4.collect()).collect()

How to mount another filesystem to a container (eg. Singularity) in a Nextflow process:

process foo{
	containerOptions{
		//the folder specified in params.inFolder is mounted to /home in the container
		//NB perhaps be careful to not to mount such a large folder
		"-B ${absolutePath('params.inFolder')}:/home"
	}
	input:
		val (in) from in_ch
	output:
		file("*.txt") into out_ch

	```
	touch file.txt

	```

}

How to store a value from a script into a nextflow channel (this hasn't been tested):

process myProcess {

   input:
      val (in) from inChannel

    output:
       shell '''
       echo "Hello world!"
       echo "This is a multiline command."
       ''' into outChannel
}

How to re-enable the bi-directional clipboard in virtualbox with a xubuntu guest:

systemctl restart vboxadd-service
#alternatively try:
sudo VBoxClient --clipboard

How to remove tabs between double quotes in a tab-separated file using a regular expression in Perl:

#This assumes you cannot have escaped quotes inside your quotes. And that your fields themselves do not contain quotes.
$line = join "", map {my $_ = $_; /^"/ && s/\t+//g; $_} $line =~ /"[^"]*"|[^"]*/g;

[Matt]

How to set an access policy in Amazon Web Services to only allow read/write access to a single bucket:

############################################################
#address: https://s3.amazonaws.com/bucket-name
#Settings for cyberduck
#server: s3.amazonaws.com
#path: /bucket-name/
############################################################

{
    "Id": "p1",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "sp1",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::bucket-name",
                "arn:aws:s3:::bucket-name/*"
            ]
        }
    ]
}

[Matt]

How to combine characters with unicode diacritical marks and print them out in Perl.

Perl code must contain:

use open qw(:std :utf8);

To combine the character with the mark you have to use the 3 digits following the 0x0 prefix in the first column of the table below. You code it in the following way:

print "A\x{302}\n";

will print out:

Â

Multiple marks can be used:

print "A\x{302}\x{32A}\n";

Diacritical Marks Table:

Position	Decimal	Name	Appearance
0x0300	768	COMBINING GRAVE ACCENT	̀
0x0301	769	COMBINING ACUTE ACCENT	́
0x0302	770	COMBINING CIRCUMFLEX ACCENT	̂
0x0303	771	COMBINING TILDE	̃
0x0304	772	COMBINING MACRON	̄
0x0305	773	COMBINING OVERLINE	̅
0x0306	774	COMBINING BREVE	̆
0x0307	775	COMBINING DOT ABOVE	̇
0x0308	776	COMBINING DIAERESIS	̈
0x0309	777	COMBINING HOOK ABOVE	̉
0x030A	778	COMBINING RING ABOVE	̊
0x030B	779	COMBINING DOUBLE ACUTE ACCENT	̋
0x030C	780	COMBINING CARON	̌
0x030D	781	COMBINING VERTICAL LINE ABOVE	̍
0x030E	782	COMBINING DOUBLE VERTICAL LINE ABOVE	̎
0x030F	783	COMBINING DOUBLE GRAVE ACCENT	̏
0x0310	784	COMBINING CANDRABINDU	̐
0x0311	785	COMBINING INVERTED BREVE	̑
0x0312	786	COMBINING TURNED COMMA ABOVE	̒
0x0313	787	COMBINING COMMA ABOVE	̓
0x0314	788	COMBINING REVERSED COMMA ABOVE	̔
0x0315	789	COMBINING COMMA ABOVE RIGHT	̕
0x0316	790	COMBINING GRAVE ACCENT BELOW	̖
0x0317	791	COMBINING ACUTE ACCENT BELOW	̗
0x0318	792	COMBINING LEFT TACK BELOW	̘
0x0319	793	COMBINING RIGHT TACK BELOW	̙
0x031A	794	COMBINING LEFT ANGLE ABOVE	̚
0x031B	795	COMBINING HORN	̛
0x031C	796	COMBINING LEFT HALF RING BELOW	̜
0x031D	797	COMBINING UP TACK BELOW	̝
0x031E	798	COMBINING DOWN TACK BELOW	̞
0x031F	799	COMBINING PLUS SIGN BELOW	̟
0x0320	800	COMBINING MINUS SIGN BELOW	̠
0x0321	801	COMBINING PALATALIZED HOOK BELOW	̡
0x0322	802	COMBINING RETROFLEX HOOK BELOW	̢
0x0323	803	COMBINING DOT BELOW	̣
0x0324	804	COMBINING DIAERESIS BELOW	̤
0x0325	805	COMBINING RING BELOW	̥
0x0326	806	COMBINING COMMA BELOW	̦
0x0327	807	COMBINING CEDILLA	̧
0x0328	808	COMBINING OGONEK	̨
0x0329	809	COMBINING VERTICAL LINE BELOW	̩
0x032A	810	COMBINING BRIDGE BELOW	̪
0x032B	811	COMBINING INVERTED DOUBLE ARCH BELOW	̫
0x032C	812	COMBINING CARON BELOW	̬
0x032D	813	COMBINING CIRCUMFLEX ACCENT BELOW	̭
0x032E	814	COMBINING BREVE BELOW	̮
0x032F	815	COMBINING INVERTED BREVE BELOW	̯
0x0330	816	COMBINING TILDE BELOW	̰
0x0331	817	COMBINING MACRON BELOW	̱
0x0332	818	COMBINING LOW LINE	̲
0x0333	819	COMBINING DOUBLE LOW LINE	̳
0x0334	820	COMBINING TILDE OVERLAY	̴
0x0335	821	COMBINING SHORT STROKE OVERLAY	̵
0x0336	822	COMBINING LONG STROKE OVERLAY	̶
0x0337	823	COMBINING SHORT SOLIDUS OVERLAY	̷
0x0338	824	COMBINING LONG SOLIDUS OVERLAY	̸
0x0339	825	COMBINING RIGHT HALF RING BELOW	̹
0x033A	826	COMBINING INVERTED BRIDGE BELOW	̺
0x033B	827	COMBINING SQUARE BELOW	̻
0x033C	828	COMBINING SEAGULL BELOW	̼
0x033D	829	COMBINING X ABOVE	̽
0x033E	830	COMBINING VERTICAL TILDE	̾
0x033F	831	COMBINING DOUBLE OVERLINE	̿
0x0340	832	COMBINING GRAVE TONE MARK	̀
0x0341	833	COMBINING ACUTE TONE MARK	́
0x0342	834	COMBINING GREEK PERISPOMENI	͂
0x0343	835	COMBINING GREEK KORONIS	̓
0x0344	836	COMBINING GREEK DIALYTIKA TONOS	̈́
0x0345	837	COMBINING GREEK YPOGEGRAMMENI	ͅ
0x0346	838	COMBINING BRIDGE ABOVE	͆
0x0347	839	COMBINING EQUALS SIGN BELOW	͇
0x0348	840	COMBINING DOUBLE VERTICAL LINE BELOW	͈
0x0349	841	COMBINING LEFT ANGLE BELOW	͉
0x034A	842	COMBINING NOT TILDE ABOVE	͊
0x034B	843	COMBINING HOMOTHETIC ABOVE	͋
0x034C	844	COMBINING ALMOST EQUAL TO ABOVE	͌
0x034D	845	COMBINING LEFT RIGHT ARROW BELOW	͍
0x034E	846	COMBINING UPWARDS ARROW BELOW	͎
0x0360	864	COMBINING DOUBLE TILDE	͠
0x0361	865	COMBINING DOUBLE INVERTED BREVE	͡
0x0362	866	COMBINING DOUBLE RIGHTWARDS ARROW BELOW	͢

[Matt]

How to change ownership of a topic/first post in NodeBB:

• Login to mongo and select the nodebb db:

mongo --port 27017
use nodebb

• Find the object containing the title of the Topic and note down the "mainPid" value of the first post (49 in the example below):

> db.objects.find({"title":"Party"},{"content":0}).pretty()

{
	"_id" : ObjectId("5ce891520ff514295e68ba97"),
	"_key" : "topic:32",
	"tid" : 32,
	"uid" : 18,
	"cid" : 19,
	"mainPid" : "49",
	"title" : "Party",
	"slug" : "32/party",
	"timestamp" : 1558745426311,
	"lastposttime" : 1559511732448,
	"postcount" : 3,
	"viewcount" : 19,
	"locked" : 0,
	"deleted" : 0,
	"pinned" : 0,
	"thumb" : "",
	"teaserPid" : 72
}
{
	"_id" : ObjectId("5cf0b04e60bb27abedd56188"),
	"_key" : "event:821",
	"type" : "post-purge",
	"uid" : 18,
	"pid" : "60",
	"ip" : "137.154.34.129",
	"tid" : 32,
	"title" : "Party",
	"timestamp" : 1559277646867,
	"eid" : 821
}

• Find the "uid" of the user you wish to change ownership to (19 in the example below):

> db.objects.find({"username":"Harry"}).pretty()

{
	"_id" : ObjectId("5cf4420b60bb27abedd56383"),
	"_key" : "user:19",
	"username" : "Harry",
	"userslug" : "harry",
	"email" : "harry@harry.com",
	"joindate" : 1559511563818,
	"lastonline" : 1559512762482,
	"picture" : "",
	"fullname" : "harry_mcboatface",
	"location" : "",
	"birthday" : "",
	"website" : "",
	"signature" : "",
	"uploadedpicture" : "",
	"profileviews" : 2,
	"reputation" : 0,
	"postcount" : 3,
	"topiccount" : 0,
	"lastposttime" : 1559512083251,
	"banned" : 0,
	"status" : "online",
	"gdpr_consent" : 0,
	"acceptTos" : 0,
	"uid" : 19,
	"ldapid" : 4028644545,
	"rss_token" : "340dc97c-0586-4ecd-8584-2241d106ce03"
}

• Use the "uid" value from above (19) to update the "cid" and "uid" values of the topic

> db.objects.update({ "title":"Party" },{$set: { "cid" : 19 }})
> db.objects.update({ "title":"Party" },{$set: { "uid" : 19 }})

• Use the "uid" value from above (19) to update the first post from above ("pid" 49)

> db.objects.update({ "pid": 49 },{$set: { "uid" : 19 }})

• If you want to be very thorough you can update the uid of the post edit history. Find all edit 'eid' values shown after running the very first command in this post :

db.objects.find({"title":"Party"},{"content":0}).pretty()

          and update their uid:

db.objects.update({ "eid": 821 },{$set: { "uid" : 19 }})

• You then change the uid of all posts in the topic by inspecting the html of each post for the uid in your browser (eg. in chrome on windows right-click the post and choose inspect from the pull down menu):

• Then use the following code in mongo to update the uid for that pid:

db.objects.update({ "pid": 299 },{$set: { "uid" : 19 }})

• You need to rebuild and restart the database using the Admin Control Panel (ACP) for the changes to take effect.

[Matt]

How to view Microsoft Word documents online in NodeBB:

[normal link](/assets/uploads/files/1559590085004-test-word-document.docx)

[viewable link](https://view.officeapps.live.com/op/view.aspx?src=https%3A%2F%2Flinks.sharezomics.com%2Fassets%2Fuploads%2Ffiles%2F1559590085004-test-word-document.docx)

normal link

viewable link

[Matt]

How to rip/copy DVD's to your hard drive:

Method 1 (NB. Had problems with out of sync voices with this one)

• Download and install VLC media player.

• Run VLC media player.

• Insert DVD.

• In VLC media player, click Media, and then click Convert / Save... The Open Media window opens.

• Tick 'No disc menus' and raise the Title number to 1

• Click Convert / Save.

• Click the additional settings button:

• Choose Ogg/Ogm encapsulation:

• Choose MPEG-4 Video codec:

• Click save and provide a Destination file name
• Click Start

Method 2 (better)

• Download and install adware free version of ImgBurn 2.5.8.0 from:
  https://www.majorgeeks.com/files/details/imgburn.html

• Select 'Create image file from disc'

• Right click on the *.ISO file and choose 'mount'

• Open the VIDEO_TS.IFO file with VLC media player

[Matt]

How to organise NodeBB sub-categories into columns with CSS:

.category-children .category-children-item {
    float: left ;
    width: 20em ;
}

[Matt]

How to install TensorFlow with GPU support on Windows 10:

https://www.pugetsystems.com/labs/hpc/How-to-Install-TensorFlow-with-GPU-Support-on-Windows-10-Without-Installing-CUDA-UPDATED-1419/

[Matt]

How to convert a docker image to a singularity image:

docker run \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v D:\host\path\where\to\output\singularity\image:/output \
  --privileged -t --rm \
  filo/docker2singularity \
  ubuntu:14.04

Replace D:\host\path\where\to\output\singularity\image with a path in the host filesystem where your Singularity image will be created. Replace ubuntu:14.04 with the docker image name you wish to convert (it will be pulled from Docker Hub if it does not exist on your host system).

You can read more and submit issues or patches at https://github.com/chrisfilo/docker2singularity

[Matt]

How to filter mapped/unmapped fastq reads:

• get all the reads where both mapped:

samtools view -F 12 bothEndsMapped.bam

• get all the reads that did not map, but whose mate mapped:

samtools view  -F 4 -f 8 onlyThisEndMapped.bam

• get all the reads that mapped, but whose mates did not:

samtools view -F 8 -f 4 onlyThatEndMapped.bam

• get all the reads where neither one mapped:

samtools view -f 12 neitherEndMapped.bam

[Matt]

How to write nodejs async/await code:

// With function declaration
async function myFn() {
  // await ...
}
// With arrow function
const myFn = async () => {
  // await ...
}

Source

[Matt]

How to display the contents of a circular JSON object in nodejs:

Method1:

		const getCircularReplacer = () => {
		  const seen = new WeakSet();
		  return (key, value) => {
		    if (typeof value === "object" && value !== null) {
		      if (seen.has(value)) {
		        return;
		      }
		      seen.add(value);
		    }
		    return value;
		  };
		};

		var string=JSON.stringify(circularJSONobject, getCircularReplacer());
		console.log(string);

Method 2:

var util = require('util'); //a core nodejs function but it has to be imported
console.log(util.inspect(circularJSONobject));

[Matt]

How to create post buttons that look like links:

css:

  .link-button {
    background: none;
    border: none;
    color: #337ab7;
    cursor: pointer;
    font-size: 19px;
    font-weight: 400;
    font-family: 'Lato', sans-serif;
  }
  .link-button:focus {
    outline: none;
  }
  .link-button:active {
    color:red;
  }
  .link-button:hover {
    text-decoration: underline;
    filter: brightness(65%);
  }

html:

<form method="post" action="https://mics.com/index" class="inline">
<button type="submit" name="submit_name" value="name" class="link-button">link display text</button>
</form>

[Matt]

How to display line numbers for pasted code in NodeBB:

This is from here.

NB. (This isn't working for me)

Put this in your custom header (ACP -> Appearance -> Custom HTML & CSS -> Custom Header):

<script>
require(['highlight'], function (hljs) {
  require([
    'https://cdnjs.cloudflare.com/ajax/libs/highlightjs-line-numbers.js/1.1.0/highlightjs-line-numbers.min.js'
  ], function () {
    $(window).on('load action:ajaxify.end', function () {
      setTimeout(function () {
        hljs.initLineNumbersOnLoad();
      }, 100);
    });
  });
});
</script>

NB. The contents of 'highlightjs-line-numbers.min.js' are:

!function(e){"use strict";function t(){"complete"===document.readyState?n():e.addEventListener("DOMContentLoaded",n)}function n(){try{var e=document.querySelectorAll("code.hljs");for(var t in e)e.hasOwnProperty(t)&&r(e[t])}catch(n){console.error("LineNumbers error: ",n)}}function r(e){if("object"==typeof e){var t=e.parentNode,n=o(t.textContent);if(n>1){for(var r="",c=0;n>c;c++)r+=c+1+"\n";var l=document.createElement("code");l.className="hljs hljs-line-numbers",l.style["float"]="left",l.textContent=r,t.insertBefore(l,e)}}}function o(e){if(0===e.length)return 0;var t=/\r\n|\r|\n/g,n=e.match(t);return n=n?n.length:0,e[e.length-1].match(t)||(n+=1),n}"undefined"==typeof e.hljs?console.error("highlight.js not detected!"):(e.hljs.initLineNumbersOnLoad=t,e.hljs.lineNumbersBlock=r)}(window);

[Matt]

How to cycle/loop through an object in javascript:

for (var key in req.body) {
    if (key in req.body) {
        console.log(req.body[key]);
    }
}

[Matt]

%(#ffffff)[How to publish a package with npm:]

*Sign in with npm login:

npm login

You’ll be prompted to enter your username, password, and email address.

*Create a folder named how-to-publish-to-npm:

mkdir how-to-publish-to-npm

*Navigate into the folder:

cd how-to-publish-to-npm

*Begin the project with the npm init command:

npm init

This command runs you through a few questions and creates a package.json file for you at the end. This package.json file contains the bare necessities you need to publish your project. (Feel free to skip questions that don’t make sense).

*The final step is to publish your package with the npm publish command:

npm publish

[Matt]

How to set permissions in linux with chmod:

chmod 600 {filespec}	You can read and write; the world can't. Good for files.
chmod 700 {filespec}	You can read, write, and execute; the world can't. Good for scripts.
chmod 644 {filespec}	You can read and write; the world can only read. Good for web pages.
chmod 640 {filespec}	You can read and write; group can read, the world can't do anything.Good for group project..
chmod 755 {filespec}	You can read, write, and execute; the world can read and execute. Good for programs you want to share, and your public_html directory.
chmod 750 {filespec}	You can read, write, and execute; the group can read and execute, the world can't do anything. Good for programs you want to share within group.

[Matt]

How to install WebODM on Windows 10 home:

• Download Docker Toolbox and install it as an administrator. (You may need to uninstall VirtualBox first.)
• Launch Docker Quickstart Terminal as an administrator. Note down the IP address shown below the whale.

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/

docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.com

• Download WebODM with Git and start it:

git clone https://github.com/OpenDroneMap/WebODM --config core.autocrlf=input --depth 1
cd WebODM
sh ./webodm.sh start

• Navigate to port 8000 of the IP address recorded in the first step:

http://192.168.99.100:8000

• It is advisable to resize the VM associated with WebODM.

cd "C:\Program Files\Oracle\VirtualBox"
./VBoxManage clonemedium disk --format VDI "C:\Users\username\.docker\machine\machines\default\disk.vmdk" "C:\Users\username\.docker\machine\machines\default\disk.vdi"

./VBoxManage modifyhd "C:\Users\username\.docker\machine\machines\default\disk.vdi" --resize 100000

After resizing you will need to use a GParted VM to resize the partitions.

• If you want to stop WebODM then use the command:

sh ./webodm.sh down

• To completely stop WebODM you will have to also stop the VirtualBox VBoxHeadless process using the Windows task manager

[Matt]

How to pipe output from a shell command to Perl to a shell command within a Perl script:

$com='samtools view -h '.$infile;
my $zom='samtools view -bS - > q64converted.bam';
open(SBAM, "$com|");
open SBOUT, "|$zom";
while (<SBAM>){
	print SBOUT "$_";
}
close(SBAM);
close SBOUT;

[Matt]

How to find out which pages are visited according to user country in Google Analytics: