Biomedical science and Mathematics

Craig Venter

Mon, 04 May 2026 00:00:00 +0000

If I don’t write it down, I never seem to find time to do it. Craig Venter passed away in April 29, it makes me feel my world become old and faraway with these excited people.

For me, it marks the end of an era.

Venter is someone I first learned about from textbooks when I studied genomics. My mentor in the US also worked at the J. Craig Venter Institute before, and he told me that he could focus on a project and not talk to anyone for a week (which I doubt), but that place seems amazing.

The most important thing I learned in last two years

Sun, 29 Mar 2026 00:00:00 +0000

What people say to you may not reveal their true character, watch how they treat others.

I finished my PhD in the last two years, but I haven’t updated my blog since 2023. It is not just too busy with research, but also about seeing how people treat others, and for me, how to manage up (apparently, I am not good at it).

Academic life is all about how to manage your time, your thoughts, your health, and your emotions. Thanks to the good environment, peers, and more experienced supervisors, I graduated.

I will keep writing, my views are my own.

How to update all R packages after installing a new version of R?

Wed, 05 Jul 2023 00:00:00 +0000

Many R users are unfamiliar with the process of updating all their R packages after upgrading to a new version of R.

Would you like to learn how to update all R pacakges in a safe and straightforward way?

The method I described below should work for all platforms, however, I didn’t use R in Windows, so I don’t know the specific directory path, I will describe the steps using a Linux system.

For example, I upgraded R from 4.2.1 to 4.3.1 recently.

As we can see in the R library folder, there are two sub-folders, one is 4.2 (/home/ylj/R/x86_64-pc-linux-gnu-library/4.2) and the other is 4.3 (/home/ylj/R/x86_64-pc-linux-gnu-library/4.3).

First, we need to copy all the packages from the 4.2 folder to the 4.3 folder. Alternatively, it is also possible to perform the reverse operation, but in that case, the folder needs to be renamed.

Then, in R terminal/Rstudio run each line below.

## load all installed packages
pkglist=data.frame(installed.packages(lib.loc="/path/to/lib")) # at here, lib.loc="/home/ylj/R/x86_64-pc-linux-gnu-library/4.3"
## update all packages
BiocManager::install(pkglist$Package,update=TRUE,ask=FALSE,checkBuilt=TRUE)
## reload new Rsession
## check if there are any packages that weren't updated
pkgupdated=data.frame(installed.packages(lib.loc="/path/to/lib"))
pkgfailed=pkgupdated[pkgupdated$Built<"4.3.1",]

For all packages in the pkgfailed. we have to install them manually.

Lifelong Learning

Sat, 01 Jul 2023 00:00:00 +0000

I have initiated a category dedicated to discussing the valuable insights gained from my supervisors, and peers, and sharing personal reflections/thoughts. This category aims to share thoughts during the ongoing, voluntary, and self-motivated pursuit of knowledge for either personal or professional reasons (lifelong learning).

In this initial post, I will continuously update and share insightful quotes from various individuals.

J: You are using only two numbers to determine your life. confusion matrix isn’t enough, let’s use AUC.

T: If this decision is hard to make, that means this decision does not matter. If this decision does not matter, then you can decide by tossing a coin.

“Must a name mean something?” Alice asks Humpty Dumpty, only to get this answer: “When I use a word… it means just what I choose it to mean – neither more nor less.”

This reminds me of a famous statement from the 1960s: The medium is the message. Here you seem to be letting Google docs or Google form/spreadsheet determine how to organise your thoughts. That doesn’t seem to me to be the right way around.

Keep good dry lab habits

Wed, 28 Apr 2021 00:00:00 +0000

I have been worked in multiple dry labs during the last 10 years. Each laboratory has its own style, but few of them training students with good dry lab habits. I learned a lot of good habits from Dr. Malay Basu’s lab. Now, I would like to share these good habits and tips when doing bioinformatics study in a dry lab. All the suggestions based on the Linux environment.

Habtis

No.1

Maintain a good documentary for all analysis. I learned this since the first week at Malay’s lab. We recommand follow the style of Rmarkdown reprot from MD Anderson Cancer Center

The report should includes:

Executive Summary (*)
- Introduction
- Data and Methods
- Results
Data Mungling
Analysis
- Step 1
- Step 2
Appendix (*)
- Working dir
- Script descriptions
- List of Figures
- List of Tables

If you want to share the report with others, it is better to generate a HTML report, if you want to see the report as a MD file on Github, you need to generate a Github MD report.

Always use sessionInfo() to print the collect information about the current R session in the bottom of the report.

No.2

Each research project should also follow a clean structure:

bin
original_data
- README - containing where the data is from
- Data in compressed form.
  1. If the raw data is huge. Then only the process data. The the processed step should be captured in the “exp” directory.
“exp” - Actual experiments. Each directory must be named as
1. YYYYMMDD-some_identifiable_info-#githubissue-INITIALS
2. each directory should link input files in in folder and generate outputfiles in out folder. The outfiles should have the name date and time appended to it.
3. The entire experiments should be done through and RMD file or single “run.sh” or a “Makefile”. There must be the RMD/MD file in each directory. You must write the executable summary and the list of figures and tables and their descriptions in the report.
“design” (?)
- Reserved for PI.
docs
- Reserved for PI.
reports
1. “index.htm”: A top level index files listing every file and few lines description of the files in the directory.
2. A list of files names matching with the each directory names of “exp”, preferebly in html format generated from RMDs

You can upload the project structure and code to any cloud storage, make a backup every day/week.

No.3

Whatever, back up the server data every week, lost data is a huge pain for every researcher.

No.4

After a project is finished. You need to review the whole project’s data and make sure every folder in the project has a document to record what you had done before, every script should also have a note to record the meaning of it.

No.5

when link a data from one sub-folder to another sub-folder in one project folder, you should use relative symbolic link ln -rs. When you need to move a project folder to other place, the data structure will not be broken.

Tips

No.1

Use tmux or screen when you want to run some program after exit the terminal session, they are usable with interactive commands. I perfer using tmux , because it has more function compare with screen. Please try to avoid to use nohup which you couldn’t control the program later.

When you have a Rstudio-server, sometime you may wish to run program via Rstudio termianl after you close the Rstudio interface, but I still suggest you run the program using tmux inside the Rstudio terminal.

The main reason for this is when the Rstudio terminal session is not reponse, you still can do interactive in tmux from other ssh session.

No.2

Always monitor the memory usage via ps aux --sort -rss | head, some parallel program may cause momory leak, you maynot know about memory leak even the program is running sucessfully.

No.3

Control+C aborts the application almost immediately while Control+Z shunts it into the background, suspended. If you shunt any application into the background, please keep in mind that the program is still waiting for your command.

No.4

Do not save the large files to /home directory if your server has limited storage for /home. Use df -h to check the size of each system disk.

Use du -sh * | sort -rh to check the size of each folder.

No.5

Better to learn one of Pipeline Building Framework and Docker, it makes everything reproducible more easily.

No.6

You need have basic knowledge of conda, and use it when run python code.

Also makesure you know where is the installed version of it. I found someone who is a su user installs miniconda on the /home directory but the all user’s environment of the conda has changed to this person’s /home version. That’s ridiculous.

No.7

Always compress dataset in to .xz or .gz format. R and Python (or other programming language) has function to read compressed datasets, you don’t need to uncompress any dataset for analysis. That’s may save a lot of space.

No.8

When generate a modified file, never cover the orginal files.

No.9

Bioinformatics is a field that has rapid changes. Stay hungry, keep learning.

Some useful R codes

Mon, 04 May 2020 00:00:00 +0000

This is an Attic to store some useful R codes, I don’t want to search them every time, so I put these codes at here.

Check packages if they are installed

check.package <- function(pkg){
  new.pkg <- pkg[!(pkg %in% installed.packages()[, "Package"])]
  if (length(new.pkg)) 
    install.packages(new.pkg, dependencies = TRUE)
  sapply(pkg, require, character.only = TRUE)
}


packages.name <- c("reshape","ggplot2","gridExtra")
check.package (packages.name)

Draw beautiful arrowhead

require(ggplot2)
require(grid)

d = seals[sample(1:nrow(seals), 100),]

ggplot(d, aes(x = long, y = lat)) +
geom_segment(aes(xend = long + delta_long/100, 
                 yend = lat + delta_lat/100),
             arrow = arrow(angle = 10,type="closed"),
             colour=c("black"),
             arrow.fill=c("red"),
             size=0.5) 

TBC

How to convert webm to gif on Linux

Mon, 04 May 2020 00:00:00 +0000

webm is an audiovisual media file format. It is primarily intended to offer a royalty-free alternative to use in the HTML5 video and the HTML5 audio elements. It has a sister project WebP for images. On Fedora system, you could use the Gnome’s embedded screencast tool to create a 30 seconds video of your screen by default. What I want is a gif animation that would be easier to be transferred and shown online.

So how to convert webm file to gif file on Linux?

The best way is using ffmpeg ¹.

ffmpeg -y -i input.webm -vf palettegen palette.png
ffmpeg -y -i input.webm -i palette.png -filter_complex paletteuse -r 10 output.gif

After that I also recommend using GNU Image Manipulation Program (GIMP) to crop off the unwanted part in the animation, it is also a way to reduce the animation size.

How to do I convert an webm (video) to a (animated) gif on the command line? ↩

working with VCF files

Thu, 09 Apr 2020 00:00:00 +0000

1.sort VCF files
2.index VCF files
3.extract vcf from a bed region
4.intersect VCF files
5.merge VCF files
6.Concatenate or combine or append VCF files
7.Change Chromosome Notation

This is a note of working with VCF files. Try to avoid use awk/sed and other linux default command. Use the professional tools!

1.sort VCF files

java -jar picard.jar SortVcf INPUT=in.vcf OUTPUT=out.vcf
java -jar picard.jar SortVcf INPUT=in.vcf.gz OUTPUT=out.vcf.gz

It will provide index file if you set the output as gz file.

2.index VCF files

bgzip genotypes.vcf && tabix -p vcf genotypes.vcf.gz

Ref:Question: Generate vcf.gz file and its index file vcf.gz.tbi

3.extract vcf from a bed region

Need have a vcf index file before extract from the vcf file. Print header line with -h.

tabix -R region.bed myvcf.gz -h > extract.vcf

Ref:Question: Extract Sub-Set Of Regions From Vcf File

4.intersect VCF files

bcftools isec  -p output_dir  A.vcf.gz B.vcf.gz

output_dir/0000.vcf.gz would be variants unique to A.vcf.gz
output_dir/0001.vcf.gz would be variants unique to B.vcf.gz
output_dir/0002.vcf.gz would be variants shared by A.vcf.gz and B.vcf.gz as represented in A.vcf.gz
output_dir/0002.vcf.gz would be variants shared by A.vcf.gz and B.vcf.gz as represented in B.vcf.gz

Ref: Question: intersect VCF files

5.merge VCF files

When we say merge VCF files, it means that all genotype of one snv/indel will be merged in single line.

A.vcf

 ##fileformat=VCFv4.1 
 ##FILTER=<ID=PASS,Description="Passed all filters">
 ##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Read Depth">
 ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
 #CHROM POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  S1  S2  S3
 10  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 0/0 0/1
 11  .   C   A   .   .   DP=3;CALLER=Samtools    GT  .   .   1/1
 12  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/0 0/0 0/0
 13  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 1/1 1/1

B.vcf

 ##fileformat=VCFv4.1 
 ##FILTER=<ID=PASS,Description="Passed all filters">
 ##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Read Depth">
 ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
 #CHROM POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  S4  S5  S6
 10  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 0/0 0/1
 11  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1  .  1/1
 12  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/0 0/0 1/1 
 13  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/0 0/0 0/0

merged.vcf

 ##fileformat=VCFv4.1 
 ##FILTER=<ID=PASS,Description="Passed all filters">
 ##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Read Depth">
 ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
 #CHROM POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  S1  S2  S3 S4  S5  S6
 10  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 0/0 0/1 0/1 0/0 0/1
 11  .   C   A   .   .   DP=3;CALLER=Samtools    GT  .   .   1/1 0/1  .  1/1
 12  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/0 0/0 0/0 0/0 0/0 1/1 
 13  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 1/1 1/1 0/0 0/0 0/0

bcftools only accept zipped vcf files. -Oz is the output format of .gz

bcftools merge *vcf.gz -Oz -o Merged.vcf.gz

Ref:Question: Best way to merge multiple VCF files

6.Concatenate or combine or append VCF files

All source files must have the same sample columns appearing in the same order. The program can be used, for example, to concatenate chromosome VCFs into one VCF, or combine a SNP VCF and an indel VCF into one. The input files must be sorted by chr and position.

A.vcf

 ##fileformat=VCFv4.1 
 ##FILTER=<ID=PASS,Description="Passed all filters">
 ##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Read Depth">
 ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
 #CHROM POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  S1  S2  S3
 10  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 0/0 0/1
 11  .   C   A   .   .   DP=3;CALLER=Samtools    GT  .   .   1/1
 12  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/0 0/0 0/0
 13  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 1/1 1/1

B.vcf

 ##fileformat=VCFv4.1
 ##FILTER=<ID=PASS,Description="Passed all filters">  
 ##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Read Depth">
 ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
 #CHROM POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  S1  S2  S3
 14  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 0/0 0/1
 15  .   C   A   .   .   DP=3;CALLER=Samtools    GT  .   .   1/1
 16  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/0 0/0 0/0
 17  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 1/1 1/1

concat.vcf

 ##fileformat=VCFv4.1
 ##FILTER=<ID=PASS,Description="Passed all filters">
 ##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Read Depth">
 ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
 #CHROM POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  S1  S2  S3
 10  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 0/0 0/1
 11  .   C   A   .   .   DP=3;CALLER=Samtools    GT  .   .   1/1
 12  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/0 0/0 0/0
 13  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 1/1 1/1
 14  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 0/0 0/1
 15  .   C   A   .   .   DP=3;CALLER=Samtools    GT  .   .   1/1
 16  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/0 0/0 0/0
 17  .   C   A   .   .   DP=3;CALLER=Samtools    GT  0/1 1/1 1/1

bcftools concat A.vcf.gz B.vcf.gz -Oz -o output.vcf.gz

Ref:Question: How to merge vcf files with different variants but same samples?

7.Change Chromosome Notation

Please remember that in the VCF header line there also have chromosome notaion, if you use awk/sed to change the notaion, you should also change it on the VCF header line.

echo "1 chr1" >> chr_name_conv.txt
echo "2 chr2" >> chr_name_conv.txt
bcftools annotate --rename-chrs chr_name_conv.txt original.vcf.gz | bgzip > rename.vcf.gz

Ref:Question: VCF files: Change Chromosome Notation

Rising of Populism

Sun, 22 Mar 2020 00:00:00 +0000

Trump started to use the term “Chinese Virus” in twitter and briefings in the White House since 17 March. It really let me feel angry, he almost did nothing to help the Americans stop the spread of coronavirus except blame China. However, he stands for Americans, it implies most of Americans may agree with his view – The coronavirus is a “Chinese Virus”. Trump’s actions are all driven by domestic politics in preparation for the 2020 election. COVID-19 is a global public health event. The United States is facing a huge pandemic prevention challenge, and the pandemic is likely to develop into a public crisis. Trump has a very clear mind on this, in order to ensure that his re-election is not affected by the COVID-19 pandemic, he needs to shape a “story” for COVID-19. “Foreign virus” is such a “story”. Its theme is: the United States is being invaded by foreign virus, the virus is imported from “other countries”, threatening the health and safety of the American public. Americans are innocent victims. Trump’s role as president is to protect the United States.

Trump wants to “up the ante”, to make the stroy more vivid by throw out the “Chinese virus”. It clearly links the virus to China. And with China-threat theory, U.S.-China confrontation being the topic of U.S. politics for the last two years, cracking down and containing China is now politically correct in Washington. China was supposed to be the central topic of the Trump 2020 election.

The use of term “Chinese virus” is an attempt to link infectious diseases to anti-Chinese sentiment and anti-Chinese politics.

Trump’s story is that the COVID-19 is coming precisely from the biggest threat - China. Even though China did not “intentionally export” the virus, its “failure to disclose” and “poor management” led to its spread to the U.S, causing serious damage to the U.S.

In this way, people unconsciously shift their attention from their own epidemic prevention and government responsibility to China. They do not blame the government for its poor epidemic prevention, but to think that China exports viruses to the world and hurts the world. American politicians can find a scapegoat for their own failings.

I think all of the Trump supporters will agree on his view. I know from my former hospital that an American physician never goes abroad, this person believes U.S. is the best country and never visits any other place in the world (because other countries are not better than the U.S.). It is real ignorance.

The appeal of the populists has grown with mounting public discontent over the status quo. In the West, many people feel left behind by technological change, the global economy, and growing inequality. Horrific incidents of terrorism generate apprehension and fear. Some are uneasy with societies that have become more ethnically, religiously and racially diverse. There is an increasing sense that governments and the elite ignore public concerns. The Dangerous Rise of Populism

Nativism, xenophobia, racism are on the rise. We couldn’t avoid facing more and more conflict between China and the West. Nobody will win in the last.

In China, there are also lots of people support for Populism, some of them believe the COVID-19 is de novo synthetic viruses, some of them believe traditional medicine is better than modern medical medicine. They believe the person who says the truth but not consistent with the government voice is the spokesperson for the West. Personal attacks via social media are more and more common.

The ideology gap between the West and China will become larger than we could think.

The economic gap between the developed countries and developing countries will become larger than we could predict. By the way, do you see many artists from South East Asia or Latin American or Africa at the virtual concert series – One World: Together at Home? Do you think those people have their voice in the world?

How to set up Samba on CentOS

Fri, 28 Feb 2020 00:00:00 +0000

Samba is the standard Windows interoperability suite of programs for Linux and Unix. People can browse a server directory like use Windows OS on their own computer. It also could avoid the scp step if you want to download and open a text file in the server, yes, you could open any files on the server and edit it like on your own computer.

Here I will introduce the steps to set up Samba on CentOS in a home network. There may be a problem if someone wants to use this Samba function on Windows 10 systems with a server that support different version of the Samba protocol. My laptop OS is Fedora, and the server OS is CentOS.

Install Samba on local computer and server, eg: yum install samba samba-client samba-client-libs.
Start the protocol on server, sudo systemctl start smb.service.
- The smbd service provides file sharing and printing services and listens on TCP ports 139 and 445. At here, for personal reason, I didn’t want to start the nmbd server, which provides NetBIOS over IP naming services to clients and listens on UDP port 137.

Set up firewall

firewall-cmd --list-services
firewall-cmd --list-ports
firewall-cmd --permanent --zone=public --add-service=samba

I want to use the /home/yulijia directory via Samba, so the Samba configuration file (/etc/samba/smb.conf) could be:
```
[homes]
 comment = Home Directories
 valid users = %S, %D%w%S
 browseable = Yes
 read only = No
 create mask = 0700
     directory mask = 0700
 inherit acls = Yes
```
Once done, please restart Samba services, systemctl restart smb.service
Connect from your local machine. Because I use Fedora, so I could open the Nautilus to add a new connect to server link (smb://192.168.0.111:139/), then connect to the server, select Registered User, enter your server username and password.

Then the files on the Samba server will be shown.

Note: the default port of Samba service is 139.

Reference:

[1] How to Install and Configure Samba on CentOS 7