Friday, March 12, 2021

Adding new disk

Start with check 

fdisk -l

you should see it. 

parted /dev/DISK
 mklabel GPT
 mkpart primary 0% 100%

Create ext4 and reduce root space

mkfs.ext4 /dev/DISKpart
tune2fs -m 0 /dev/DISKpart

Tuesday, October 20, 2020

New R package -- useful notes

GOAL: To create a new R package designed for CRAN and having code repository maintained by git.

Just from the beginning, there are many requirements that you should fulfill for CRAN submission, you can expect that it will take two or more rounds of the submission, just you should understand that all checks matters :-) but then https://cran.r-project.org/web/packages/gawdis

* Prepare files with R functions that should create a package (ideally one big function in one file, or couple of similar goal functions in one file).
* Choose the name, here I will mostly use gawdis (you can see all codes on https://github.com/pavel-fibich/gawdis ).
* [RStudio] Start with New project -> New directory -> R package in RStudio (check create git repository and choose files with R functions). 
* [github] Create the same New repository on GitHub (login, create repository).
* [RStudio - git terminal] Connect your local repository to github repo 

 git remote add origin https://github.com/pavel-fibich/gawdis 

* Now you can commit/pull to github directly in RStudio. 

Because for each function and data you should have help page, it is useful to let roxygen2 (see vignettes https://cran.r-project.org/web/packages/roxygen2/vignettes/roxygen2.html) generate help pages of the functions. Iit is quite easy and you do not need to know any Latex commands, you just annotate head of R file with #' and roxygen2 will generate Latex help for you). It done by direct calling  

 roxygen2::roxygenise() 

Example of annotated R file, the header looks like (the last line is the R code of the function gawdis):

#' @title gawdis function
#'
#' @description \code{gawdis()}, is an extension of the function \code{gowdis}, ...
#'
#' @param x Matrix or data frame containing the variables..
#' @param W Vector listing the weights for the variables in x...
#'
#' @usage
#' gawdis(x,W = NULL, ....
#'
#' @keywords gawdis gowdis
#' @return  An object of class dist ...
#' @references  de Bello, F. et al. (2021) Towards a more balanced ....
#'
#' Gower, J. C. (1971) A general coefficient of similarity and ....
#'
#' @seealso gowdis() from FD package.
#' @examples
#' library(FD) # input data ...
#' x<-1
gawdis<-function(....

After the call of roxygenese() new .Rd files with help appear in man folder. 

You should also have DESCRIPTION in good shape, check https://cran.r-project.org/doc/manuals/r-devel/R-exts.html#The-DESCRIPTION-file

To create vignettes, you create vignettes folder with some R markdown code, that is later build for you automatically.

Building of package is easy in RStudio (you just choose Build - More - Build from source). But first, you should check it, e.g. by more strict option --as-cran, you can use RStudio too and/or check already built package  

 R CMD check --as-cran --no-clean packagename_0.1.0.tar.gz 

It will create also pdf documentation, you should check it too. But before the submission to CRAN, you must go through checking on devel version of R (more notes on https://cran.r-project.org/web/packages/submission_checklist.html). You can also use automatic checking on devel version of R directly on command line (or use web upload):

 devtools::check_win_devel() 

and you will get email in 30 minutes about the building and checks. Just to know the policy, CRAN is quite strict on checks, you should pass all of them, there are just few exceptions (eg. new package will have some warning), but you should be sure that they are not your mistake or laziness.

Mostly all the time I have problem with the length of examples (they were to long), to reduce it, you can put them inside "\donttest{code}", they will appear in help, but they are not run in checks.

If you succeed in publication of manuscript using your package, add it to references (of the functions and in DESCRIPTION file) and submit new version of your Rpackage.


Monday, September 25, 2017

postgres life

Basically we start with
su postgres
psql galaxy
where galaxy is the name of databse.

Basic psql help
\?
List of tables
\dt
definition of the table galaxy_user
\d galaxy_user

To create automatic backup (after -U is the username)
pg_dump -U galaxy galaxy > galaxy.db
To fill database from the backup
psql galaxy < galaxy.db

Script for running in the crontab #!/bin/bash BACKUP_DIR=/home/galaxyelixir/storage/dbbackups
DAYS_TO_KEEP=30
FILE_SUFFIX=_pg_backup_galaxielixir.sql
DATABASE=galaxy
FILE=`date +"%Y%m%d"`${FILE_SUFFIX}
OUTPUT_FILE=${BACKUP_DIR}/${FILE} pg_dump ${DATABASE} -F p -f ${OUTPUT_FILE}
gzip $OUTPUT_FILE
# prune old backups
find $BACKUP_DIR -maxdepth 1 -mtime +$DAYS_TO_KEEP -name "*${FILE_SUFFIX}.gz" -exec rm -rf '{}' ';'

crontab line for making backup twice a week
5 * * * 3,6 /home/galaxyelixir/storage/dbbackups/backupPostGre.sh

Tuesday, December 11, 2012

Simple setting of MySQL server for user's prefix

Few MySQL commands that I need:

  • connect to database
    mysql --user=root mysql -p
  • create db user 'paja' for connecting to localhost and also from outside
    CREATE USER 'paja'@'localhost' IDENTIFIED BY 'paja-password';
    CREATE USER 'paja'@'%' IDENTIFIED BY 'paja-password'; 
  • grant privileges to create databases with prefix 'paja_'
    GRANT ALL PRIVILEGES ON `paja\_%` . * TO 'paja'@'localhost';
    GRANT ALL PRIVILEGES ON `paja\_%` . * TO 'paja'@'%'
    ;
  • modify /etc/mysql/my.cnf by comment line
    bind-address           = 127.0.0.1
    that makes MySQL server listening only on localhost
  • test connection from localhost and from different host
    mysql --user=paja -h localhost -p
    mysql --user=paja -h myhost.address -p

Friday, September 7, 2012

Perl modules

You have already met this
Can't locate PerlIO/gzip.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.10.0 /usr/local/share/perl/5.10.0 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at ...
because sometimes you have to install perl modules in non-standard folders (e.g. you are not root at the machine). How to solve it?

There is guide how to install perl module to specified folder
http://modperlbook.org/html/3-9-1-Installing-Perl-Modules-into-a-Nonstandard-Directory.html
In few steps:

perl Makefile.PL PREFIX=/your/folder
make
make install


  • export variable PERL5LIB to /your/folder/lib/perl/5.10.1
Now it should work.

To see what is in your @INC, run

perl -V
To test your module run
perl -Mmodule_name -e ''

Wednesday, December 14, 2011

disk information and format

It is quote common that you need information about disk. There are many commands (sometimes distribution depended) that can do that. Try following:

lshw -class disk
smartctl -i /dev/sda
hdparm -i /dev/sda
hwinfo --disk
ls /dev/disk/by-id
cat /proc/partitions

cat /proc/mdstat

To add new disk you just need

fdisk -l
parted /dev/sdb
(parted) mklabel GPT                                                      
(parted) mkpart primary 0% 100%                                           
(parted) quit
mkfs.ext4 /dev/sdb1
tune2fs -m 0 /dev/sdb1

Add line to /etc/fstab

Friday, January 28, 2011

identify problem

If something does not work first look at logs :
cat /var/log/syslog
dmesg


But there are not only these logs (see /var/log folder).

For hardware identify what was determined :
lsusb
lspci
lshal
lshw


There you easily find vendor or identification for hardware.

For software (run by command) try:
strace command


or use verbose output of command, often by:
command --verbose



For open files, see:
lsof

Mentioned utilities have many options, so try arguments. Use keywords from previous outputs to feed goog..

Wednesday, December 8, 2010

playing with linear raid and ext4

The goal is to create linear raid with ext4 array from 4 disks. Than add 5. disk and verify data.

Create linear raid from 4 disks
$ sudo mdadm -v --create /dev/md0 -l linear -n 4 /dev/sdb /dev/sdc /dev/sdd /dev/sde

Make ext4fs
$ sudo mkfs.ext4 /dev/md0

Mount array
$ sudo mkdir array
$ sudo mount /dev/md0 /media/array


Information about array
$ sudo mdadm --detail /dev/md0
$ cat /proc/mdstat


Set the system recognize the raid after the reboot
$ sudo echo 'DEVICES /dev/sdb /dev/sdc /dev/sdd /dev/sde' >> /etc/mdadm/mdadm.conf
$ sudo mdadm --detail --scan >> /etc/mdadm.conf


Upload test data, make md5 checksum
$ cd /media/array/
$ wget http://cdimage.debian.org/debian-cd/5.0.7/amd64/iso-cd/debian-507-amd64-netinst.iso
$ md5sum debian-507-amd64-netinst.iso
971ddace926fc3f7765f595da5cce223 debian-507-amd64-netinst.iso


Add 5. disk and resize filesystem
$ mdadm --grow --add /dev/md0 /dev/sdf
$ sudo resize2fs /dev/md0


Add device at /etc/mdadm/mdadm.conf and rescan (change list line)

Compare checksum
$ md5sum debian-507-amd64-netinst.iso
971ddace926fc3f7765f595da5cce223 debian-507-amd64-netinst.iso


Checksums are the same.

Add line to /etc/fstab and test reboot.

Sometimes there are problems with assemble, look in

cat /proc/mdstat


if there are not some dummy traits from one disk, that you can stop by

mdadm --stop /dev/md_d0


See also wiki

Monday, December 6, 2010

esxi managing from console

See also: Robert Chase page

It is easy to use console of ESXi server and do some managing from it.


Get all virtual machines
vim-cmd vmsvc/getallvms

Power off/on/reboot virtual machine with id 368
vim-cmd vmsvc/power.off 368
vim-cmd vmsvc/power.on 368
vim-cmd vmsvc/power.reboot 368


Get information about virtual machine with id 368
vim-cmd vmsvc/get.summary 368

Delete all snapshots
vim-cmd vmsvc/snapshot.removeall 368

Refresh datastore
esxcfg-nas -r

Delete and add nfs datastore back
esxcfg-nas -d back
esxcfg-nas -a back --host IPADDRESS --share DIRECTORY


Running processes
esxtop

Console screen
dcui

Wednesday, September 15, 2010

apache2 modules

List of loaded modules:
apache2ctl -t -D DUMP_MODULES

List of compiled modules:
apache2 -l

Apache2 version:
apache2 -v

Disable module:
a2dismod module_name

Enable module:
a2enmod module_name

Tuesday, June 15, 2010

second postfix instance with auth smtp

See advosys.ca help.ubuntu.com adomas.org postfix.org

Copy config dir
cp -rp /etc/postfix /etc/postfix-out

Create file /etc/postfix-out/sasl/smtpd.conf with
pwcheck_method: saslauthd
mech_list: PLAIN LOGIN

Note, that previous plaintext mechanism send credentials unencrypted. Therefore TLS-encrypted SMTP session is used.

Add following lines to /etc/postfix-out/main.cf
queue_directory = /var/spool/postfix-out
data_directory = /var/lib/postfix-out


Create and copy queue and data directories
mkdir /var/spool/postfix-out
postfix -c /etc/postfix-out check
cp -rp /var/lib/postfix /var/lib/postfix-out
cp -rp /var/spool/postfix/etc /var/spool/postfix-out
cp -rp /var/spool/postfix/usr /var/spool/postfix-out
cp -rp /var/spool/postfix/lib /var/spool/postfix-out


Add following line to /etc/postfix/main.cf
alternate_config_directories = /etc/postfix-out

Modify port no. for smtp at /etc/postfix-out/master.cf

1025 inet n - - - - smtpd


Test start
postfix -c /etc/postfix-out start

Install sasl and mkdir var folder
apt-get install libsasl2-2 sasl2-bin


Modify /etc/init.d/saslauthd
PIDFILE=/var/spool/postfix-out/var/run/saslauthd/saslauthd.pid

Modify /etc/default/saslauthd
START=yes
PWDIR="/var/spool/postfix-out/var/run/saslauthd"
PARAMS="-m ${PWDIR}"
PIDFILE="${PWDIR}/saslauthd.pid"
OPTIONS="-c -m /var/spool/postfix-out/var/run/saslauthd"


Create dirs for sasl in chroot postfix
dpkg-statoverride --force --update --add root sasl 755 /var/spool/postfix-out/var/run/saslauthd

Add following lines to /etc/postfix-out/main.cf
smtpd_sasl_auth_enable = yes
broken_sasl_auth_clients = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_path = smtpd


Restart all services
/etc/init.d/saslauthd restart
/etc/init.d/postfix restart
postfix -c /etc/postfix-out stop
postfix -c /etc/postfix-out start


Test by sasl
testsaslauthd -u USERNAME -p PASSWORD -f /var/spool/postfix-out/var/run/saslauthd/mux

Test by telnet
telnet SERVER_ADDRESS 1025
Trying IPADDRESS...
Connected to SERVER_NAME.
Escape character is '^]'.
220 SERVERNAME ESMTP Postfix (Ubuntu)
ehlo localhost
250-SERVERNAME
250-PIPELINING
250-SIZE 10240000
250-ETRN
250-STARTTLS
250-AUTH PLAIN LOGIN
250-AUTH=PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN


Create startup script
cp /etc/init.d/postfix /etc/init.d/postfix-out

And replace pairs in new file
/etc/postfix by /etc/postfix-out
postfix by postfix -c /etc/postfix-out
/var/spool/postfix by /var/spool/postfix-out
postconf by postconf -c /etc/postfix-out

Now make symlinks in rc.d to /etc/init.d/postfix-out, e.g.
ln -s /etc/init.d/postfix-out /etc/rc6.d/K01postfix-out

Saturday, June 5, 2010

Font Arial in R

See stat.ethz.ch

Install fonts

sudo apt-get install ubuntu-restricted-extras


In console

ttf2afm /usr/share/fonts/msttcorefonts/arial.ttf > ~/afont/arial.afm
ttf2afm /usr/share/fonts/msttcorefonts/ariali.ttf > ~/afont/ariali.afm
ttf2afm /usr/share/fonts/msttcorefonts/arialbd.ttf > ~/afont/arialbd.afm
ttf2afm /usr/share/fonts/msttcorefonts/arialbi.ttf > ~/afont/arialbi.afm


In R

postscript("fa1x.eps",
family=c("/home/pvl/afont/arial.afm","/home/pvl/afont/arialbd.afm","/home/pvl/afont/ariali.afm","/home/pvl/afont/arialbi.afm"),
horizontal=FALSE, height=4.5, width=7.5, paper = "special", pointsize=4)

Friday, April 9, 2010

eduroam throught ppp

See general link

Choose NAME of connection, VPNSERVER of your institution and your LOGIN
$ cat /etc/ppp/peers/NAME
pty "pptp VPNSERVER --nolaunchpppd"
name LOGIN
remotename PPP
require-mppe-128
file /etc/ppp/options.pptp
ipparam NAME


Choose LOGIN and PASSWORD
$ cat /etc/ppp/chap-secrets
# client server secret IP addresses
LOGIN PPP PASSWORD *


Now you simply run
pppd call NAME
route add -net ADDRESS_OF_NETWORK netmask 255.255.0.0 dev ppp0

Wednesday, March 31, 2010

ssl certificates

Generate request (aliases could be added later)
keytool -genkey -keyalg RSA -keysize 2048 -keystore serverkey.pem -validity 1100 -dname "CN=SERVERNAME, O=Jihoceska univerzita v Ceskych Budejovicich, C=CZ" -keypass PASSWORD -alias SERVERALIAS

keytool -certreq -keyalg RSA -file request.csr -keystore serverkey.pem -alias SERVERALIAS


Submit request at http://tcs.cesnet.cz/req and download keys.

Generate no password key
openssl rsa -in serverkey.pem -out serverkey_nopass.pem


Copy keys
cp serverkey_nopass.pem /etc/ssl/private/
cp servercert.pem /etc/ssl/private/


Edit apache2 conf /etc/apache2/sites-enabled/000-default-ssl
SSLEngine on
SSLCertificateFile /etc/ssl/certs/servercert.pem
SSLCertificateKeyFile /etc/ssl/private/serverkey_nopass.pem

Saturday, November 28, 2009

alpine

Store password:
touch .pine-passfile

IMAP setting from .pinerc file:
inbox-path={IMAPSERVER/ssl/user=USERNAME/novalidate-cert}INBOX
default-fcc={IMAPSERVER/ssl/user=USERNAME/novalidate-cert}sent-mail
feature-list=enable-incoming-folders,
enable-incoming-folders-checking,
incoming-checking-include-total,
save-will-not-delete
folder-collections=mail/[],
"IMAPSERVER" {IMAPSERVER/user=USERNAME/ssl/novalidate-cert}[],
"gmail" {imap.gmail.com/ssl/user=USER@gmail.com}[]

Tuesday, November 10, 2009

migration openwebmail auth_unix to auth_pam

Motivation: Create new server with openwebmail and authentification to Active Directory (AD). Old server has local hand-made copies of accounts from AD. We need to migrate inboxes, ~/mail/, ~/.openwebmail/ and aliases.

Solution: We used ubuntu server 9.10, likewise and openwebmail 2.53.

Install Ubuntu server 9.10 and checked mail server and ssh server. Setup interfaces. Install few packages:
sudo aptitude install likewise-open libauthen-pam-perl libconvert-asn1-perl libmd5-perl libnet-ldap-perl perl-suid wwwconfig-common libpg-perl apache2 libdbd-mysql-perl alpine ispell iczech language-support-cs language-pack-cs-base gcc linux-headers-$(uname -r) vlan nmap mc

Join domain with likewise (see ubuntu forum), not necessary as Administrator (user):
sudo domainjoin-cli join DOMAIN user
test join:
sudo domainjoin-cli query
Many problems appear due to setting of /etc/resolv.conf, you should choose some AD machine as nameserver

Setup apache (enable ssl):
sudo a2enmod ssl
sudo /etc/init.d/apache2 restart
sudo ln -s /etc/apache2/sites-available/default-ssl /etc/apache2/sites-enabled/000-default-ssl
sudo /etc/init.d/apache2 restart


Setup likewise, enable logging:
sudo vim /etc/samba/lwiauthd.conf
add lines
winbind use default domain = yes
log level = 10


Wget, install and setup openwebmail:
wget http://openwebmail.org/openwebmail/download/debian/owm2.53-2.deb
sudo dpkg -i owm2.53-2.deb
sudo mv /var/www/openwebmail/redirect.html /var/www/index.html
sudo vim /etc/openwebmail/openwebmail.conf

* change lines to
domainnames DOMAIN
auth_module auth_pam.pl
allowed_autologinip all

* comment lines
default_style
forced_moveoldmsgfrominbox yes

* add lines
create_syshomedir yes
quota_module quota_du.pl
quota_limit 180000
quota_treshold 200000
default_language cz
default_moveoldmsgfrominbox no
spellcheck /usr/bin/ispell -a -S -w "-" -d @@@DICTIONARY@@@ -p @@@PDICNAME@@@
spellcheck_pdicname .ispell_words
spellcheck_dictionaries czech, english, american


Repair utf-8 use in perl (see acatysmoof):
sudo vim /usr/share/perl/5.10.0/CGI.pm
change line
my $utf8 = $charset eq 'utf-8';
to
my $utf8 = 0;

Setup postfix:
sudo vim /etc/postfix/main.cf
add line:
mydomain = DOMAIN
restart
sudo /etc/init.d/postfix restart

Move script:
root@cattom1:/home/pvl# cat ./moveusers.sh
#!/bin/sh
UCTY="movelist.txt"
scp -P 22 root@OLDSERVER:/etc/aliases ./old_aliases
cat $UCTY | while read LINE; do
# home move
scp -r -P 22 root@OLDSERVER:/home/$LINE /home/DOMAIN/
cat old_aliases | grep $LINE >> /etc/aliases
chown -R $LINE:domain^users /home/DOMAIN/$LINE
# inboxes
scp -P 22 root@OLDSERVER:/home/mail/$LINE /var/mail/
chown $LINE:mail /var/mail/$LINE
done


Alternativities: diretly to ldap developer.novel

Tuesday, November 3, 2009

vmware server 2.0.1 in ubuntu 9.10

See: blog.mymediasystem.net

Try reinstall from source, and after fail of installation (vmware-install.pl) copy
patch to the instalation directory (vmware-server-distrib/), untar and run patch. Run

sudo rm -r /usr/lib/vmware/modules/binary

now you can accomplish installation by

sudo /usr/bin/vmware-config.pl

Sometimes webaccess do not work so restart server by:

sudo killall -9 vmware-hostd
sudo /etc/init.d/vmware-mgmt restart


For new installation of 2.0.2 use http://radu.cotescu.com/2009/10/30/how-to-install-vmware-server-2-0-x-on-ubuntu-9-10-karmic-koala/

Friday, October 9, 2009

backup with tape

See tldp or ubuntuforums

Backup using tar:
tar -zcvpf /archive/full-backup-`date '+%d-%B-%Y'`.tar.gz --directory / --exclude=mnt --exclude=proc --exclude=var/spool/squid .

Restore:
tar xvpfz full-backup-date-XXX.tar.gz -C /

Rewind of tape (before all operations):
mt -f /dev/st0 rewind

List of files (tar.gziped archive) on tape:
tar -tzf /dev/st0

Extracting of directory www from archive to actual directory:
tar -xzf /dev/nst0 www

Tuesday, October 6, 2009

archiving

See redhat

Tar gzip folder:
tar czvf folder.tar.gz folder/
Un-tar zip folder:
tar -xvf folder.tar.gz
Un-tar file from gzip :
tar -xvf folder.tar.gz path/in/archive/file
List of tar.gz file :
tar -tvf folder.tar.gz


Arj folder:
arj ac folder.arj folder/
Un-arj folder:
arj e folder.arj

Zip folder:
zip -r folder.zip folder/
Un-zip folder:
unzip folder.zip

Friday, October 2, 2009

quota

See redhat or linuxhelp

Install quota:
aptitude install quota quotatool

Add usrquota,grpquota to your /etc/fstab for some volume, eg.
/home /home ext3 defaults,usrquota,grpquota 1 2

Remount your volume
mount -o remount /home

Check quota
quotacheck -vagu

Turn quota on
quotaon -vagum

Set grace period
edquota -t

Modify user quota
edquota -u user

or by (200M is soft, 250 is hard limit for user new in /home):
quotatool -u new -bq 200M -l '250 Mb' /home

Copy user quota from old to new user
edquota -p old new

List of quotas on volume
repquota /home

List of all quotas
repquota -a