Tuesday, October 20, 2020

New R package -- useful notes

GOAL: To create a new R package designed for CRAN and having code repository maintained by git.

Just from the beginning, there are many requirements that you should fulfill for CRAN submission, you can expect that it will take two or more rounds of the submission, just you should understand that all checks matters :-) but then https://cran.r-project.org/web/packages/gawdis

* Prepare files with R functions that should create a package (ideally one big function in one file, or couple of similar goal functions in one file).
* Choose the name, here I will mostly use gawdis (you can see all codes on https://github.com/pavel-fibich/gawdis ).
* [RStudio] Start with New project -> New directory -> R package in RStudio (check create git repository and choose files with R functions). 
* [github] Create the same New repository on GitHub (login, create repository).
* [RStudio - git terminal] Connect your local repository to github repo 

 git remote add origin https://github.com/pavel-fibich/gawdis 

* Now you can commit/pull to github directly in RStudio. 

Because for each function and data you should have help page, it is useful to let roxygen2 (see vignettes https://cran.r-project.org/web/packages/roxygen2/vignettes/roxygen2.html) generate help pages of the functions. Iit is quite easy and you do not need to know any Latex commands, you just annotate head of R file with #' and roxygen2 will generate Latex help for you). It done by direct calling  

 roxygen2::roxygenise() 

Example of annotated R file, the header looks like (the last line is the R code of the function gawdis):

#' @title gawdis function
#'
#' @description \code{gawdis()}, is an extension of the function \code{gowdis}, ...
#'
#' @param x Matrix or data frame containing the variables..
#' @param W Vector listing the weights for the variables in x...
#'
#' @usage
#' gawdis(x,W = NULL, ....
#'
#' @keywords gawdis gowdis
#' @return  An object of class dist ...
#' @references  de Bello, F. et al. (2021) Towards a more balanced ....
#'
#' Gower, J. C. (1971) A general coefficient of similarity and ....
#'
#' @seealso gowdis() from FD package.
#' @examples
#' library(FD) # input data ...
#' x<-1
gawdis<-function(....

After the call of roxygenese() new .Rd files with help appear in man folder. 

You should also have DESCRIPTION in good shape, check https://cran.r-project.org/doc/manuals/r-devel/R-exts.html#The-DESCRIPTION-file

To create vignettes, you create vignettes folder with some R markdown code, that is later build for you automatically.

Building of package is easy in RStudio (you just choose Build - More - Build from source). But first, you should check it, e.g. by more strict option --as-cran, you can use RStudio too and/or check already built package  

 R CMD check --as-cran --no-clean packagename_0.1.0.tar.gz 

It will create also pdf documentation, you should check it too. But before the submission to CRAN, you must go through checking on devel version of R (more notes on https://cran.r-project.org/web/packages/submission_checklist.html). You can also use automatic checking on devel version of R directly on command line (or use web upload):

 devtools::check_win_devel() 

and you will get email in 30 minutes about the building and checks. Just to know the policy, CRAN is quite strict on checks, you should pass all of them, there are just few exceptions (eg. new package will have some warning), but you should be sure that they are not your mistake or laziness.

Mostly all the time I have problem with the length of examples (they were to long), to reduce it, you can put them inside "\donttest{code}", they will appear in help, but they are not run in checks.

If you succeed in publication of manuscript using your package, add it to references (of the functions and in DESCRIPTION file) and submit new version of your Rpackage.


Monday, September 25, 2017

postgres life

Basically we start with
su postgres
psql galaxy
where galaxy is the name of databse.

Basic psql help
\?
List of tables
\dt
definition of the table galaxy_user
\d galaxy_user

To create automatic backup (after -U is the username)
pg_dump -U galaxy galaxy > galaxy.db
To fill database from the backup
psql galaxy < galaxy.db

Script for running in the crontab #!/bin/bash BACKUP_DIR=/home/galaxyelixir/storage/dbbackups
DAYS_TO_KEEP=30
FILE_SUFFIX=_pg_backup_galaxielixir.sql
DATABASE=galaxy
FILE=`date +"%Y%m%d"`${FILE_SUFFIX}
OUTPUT_FILE=${BACKUP_DIR}/${FILE} pg_dump ${DATABASE} -F p -f ${OUTPUT_FILE}
gzip $OUTPUT_FILE
# prune old backups
find $BACKUP_DIR -maxdepth 1 -mtime +$DAYS_TO_KEEP -name "*${FILE_SUFFIX}.gz" -exec rm -rf '{}' ';'

crontab line for making backup twice a week
5 * * * 3,6 /home/galaxyelixir/storage/dbbackups/backupPostGre.sh

Tuesday, December 11, 2012

Simple setting of MySQL server for user's prefix

Few MySQL commands that I need:

  • connect to database
    mysql --user=root mysql -p
  • create db user 'paja' for connecting to localhost and also from outside
    CREATE USER 'paja'@'localhost' IDENTIFIED BY 'paja-password';
    CREATE USER 'paja'@'%' IDENTIFIED BY 'paja-password'; 
  • grant privileges to create databases with prefix 'paja_'
    GRANT ALL PRIVILEGES ON `paja\_%` . * TO 'paja'@'localhost';
    GRANT ALL PRIVILEGES ON `paja\_%` . * TO 'paja'@'%'
    ;
  • modify /etc/mysql/my.cnf by comment line
    bind-address           = 127.0.0.1
    that makes MySQL server listening only on localhost
  • test connection from localhost and from different host
    mysql --user=paja -h localhost -p
    mysql --user=paja -h myhost.address -p

Friday, September 7, 2012

Perl modules

You have already met this
Can't locate PerlIO/gzip.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.10.0 /usr/local/share/perl/5.10.0 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at ...
because sometimes you have to install perl modules in non-standard folders (e.g. you are not root at the machine). How to solve it?

There is guide how to install perl module to specified folder
http://modperlbook.org/html/3-9-1-Installing-Perl-Modules-into-a-Nonstandard-Directory.html
In few steps:

perl Makefile.PL PREFIX=/your/folder
make
make install


  • export variable PERL5LIB to /your/folder/lib/perl/5.10.1
Now it should work.

To see what is in your @INC, run

perl -V
To test your module run
perl -Mmodule_name -e ''

Wednesday, December 14, 2011

disk information and format

It is quote common that you need information about disk. There are many commands (sometimes distribution depended) that can do that. Try following:

lshw -class disk
smartctl -i /dev/sda
hdparm -i /dev/sda
hwinfo --disk
ls /dev/disk/by-id
cat /proc/partitions

cat /proc/mdstat

To add new disk you just need

fdisk -l
parted /dev/sdb
(parted) mklabel GPT                                                      
(parted) mkpart primary 0% 100%                                           
(parted) quit
mkfs.ext4 /dev/sdb1
tune2fs -m 0 /dev/sdb1

Add line to /etc/fstab

Friday, January 28, 2011

identify problem

If something does not work first look at logs :
cat /var/log/syslog
dmesg


But there are not only these logs (see /var/log folder).

For hardware identify what was determined :
lsusb
lspci
lshal
lshw


There you easily find vendor or identification for hardware.

For software (run by command) try:
strace command


or use verbose output of command, often by:
command --verbose



For open files, see:
lsof

Mentioned utilities have many options, so try arguments. Use keywords from previous outputs to feed goog..

Wednesday, December 8, 2010

playing with linear raid and ext4

The goal is to create linear raid with ext4 array from 4 disks. Than add 5. disk and verify data.

Create linear raid from 4 disks
$ sudo mdadm -v --create /dev/md0 -l linear -n 4 /dev/sdb /dev/sdc /dev/sdd /dev/sde

Make ext4fs
$ sudo mkfs.ext4 /dev/md0

Mount array
$ sudo mkdir array
$ sudo mount /dev/md0 /media/array


Information about array
$ sudo mdadm --detail /dev/md0
$ cat /proc/mdstat


Set the system recognize the raid after the reboot
$ sudo echo 'DEVICES /dev/sdb /dev/sdc /dev/sdd /dev/sde' >> /etc/mdadm/mdadm.conf
$ sudo mdadm --detail --scan >> /etc/mdadm.conf


Upload test data, make md5 checksum
$ cd /media/array/
$ wget http://cdimage.debian.org/debian-cd/5.0.7/amd64/iso-cd/debian-507-amd64-netinst.iso
$ md5sum debian-507-amd64-netinst.iso
971ddace926fc3f7765f595da5cce223 debian-507-amd64-netinst.iso


Add 5. disk and resize filesystem
$ mdadm --grow --add /dev/md0 /dev/sdf
$ sudo resize2fs /dev/md0


Add device at /etc/mdadm/mdadm.conf and rescan (change list line)

Compare checksum
$ md5sum debian-507-amd64-netinst.iso
971ddace926fc3f7765f595da5cce223 debian-507-amd64-netinst.iso


Checksums are the same.

Add line to /etc/fstab and test reboot.

Sometimes there are problems with assemble, look in

cat /proc/mdstat


if there are not some dummy traits from one disk, that you can stop by

mdadm --stop /dev/md_d0


See also wiki