Sunday, 30 March 2014

Storages

It came to my attention that the storage and protocols used nowadays were not that quite understandably presented. So I took my time to gather some material on the subject and did a summary from what I found, to see the original documents please see the reference in the bottŠ¾m.


 SANs are primarily used to enhance storage devices, such as disk arraystape libraries, and optical jukeboxes, accessible toservers so that the devices appear like locally attached devices to the operating systemSAN does not provide file abstraction, only block-level operations. However, file systems built on top of SANs do provide file-level access, and are known as SAN filesystems or shared disk file systems.

NAS vs SAN
The primary difference between NAS and SAN solutions is the type of access protocol. NAS protocols such as NFS and CiFS provide shared file level access to storage resources. The management of the file system resides with the NAS device. SAN protocols such as iSCSI and fibre channel provide block level access to storage resources. Block level devices are accessed by servers via the SAN, and the servers manage the file system.

  • NAS devices typically leverage existing IP networks for connectivity, enabling companies to reduce the price of entry for access to shared storage.
  • The RAID and clustering capabilities inherent to modern enterprise NAS devices offer greatly improved availability when compared with traditional direct attached storage.
  • Because NAS devices control the file system, they offer increased flexibility when using advanced storage functionality such as snapshots.
  • With 10GE connectivity, NAS devices can offer performance on par with many currently installed fibre channel SANs

Benefits of NAS
DAS is optimized for single, isolated processors and low initial cost.
DAS second hard disk is today most advisably connected as an external unit, or what is sometimes now known as a "DAS" or direct attached storage drive. DAS external hard disks connect via a USBfirewire or an E-SATA interface (see the hardware section), with USB being the most common.

On servers and high-end PC workstations (such as those used for high-end video editing), at least two hard disks are often linked together using a technology called RAID. This stands for "redundant array of independent disks" (or sometimes "redundant array of inexpensive drives"), and stores the data in each user volume on multiple physical drives.

SAN is optimized for performance and scalability. Some of the major potential 
benefits include support for high-speed Fibre Channel media which is optimized 
for storage traffic, managing multiple disk and tape devices as a shared pool 
with a single point of control, specialized backup facilities that can reduce 
server and LAN utilization and wide industry support.

NAS is optimized for ease-of-management and file sharing using lower-cost 
Ethernet-based networks. Installation is relatively quick, and storage capacity is 
automatically assigned to users on demand.

NAS gateways are optimized to provide NAS benefits with more flexibility 
in selecting the disk storage than offered by a conventional NAS device. 
Gateways can also protect and enhance the value of installed disk systems.

Despite their differences, SAN and NAS are not mutually exclusive, and may be combined in multi-protocol or unified storage arrays, offering both file-level protocols (NAS) and block-level protocols (SAN) from the same system. The best of both worlds!

Fibre Channel Protocol (FCP) is a transport protocol (similar to TCP used in IP networks) that predominantly transports SCSI commands over Fibre Channel networks.[1][2]

Fibre Channel, or FC, is a high-speed network technology (commonly running at 2-, 4-, 8- and 16-gigabit per second rates) primarily used to connect computer data storage.[1][2] 



RAID
RAID 0 and RAID 1 image
RAID 5 and RAID 10 image

Many possible RAID configurations are available. The first is called "RAID 0". This divides or "strips" the data in a storage volume across two or more disks, with half of each file written to one disk, and half to another. This improves overall read/write performance without sacrificing capacity. So, for example (as shown above), two 1TB drives may be linked to form a 2TB array. Because this virtual volume is faster than either of its component disks, RAID 0 is common used on video editing workstations.
In contrast to RAID 0, "RAID 1" is primarily intended to protect data against hardware failure. Here data is duplicated or "mirrored" across two or more disks. The data redundancy so created means that if one physical drive fails there is still a complete copy of its contents on another drive. However, this does mean that drive capacity is sacrificed. For example (as shown above), a 1TB RAID 1 volume requires two 1TB disks. While data write performance is not improved by using RAID 1, data read times are increased as multiple files can be accessed simultaneously from different physical drives.

If more than two drives are used, several other configurations become possible. For example, using three of more drives, "RAID 5" strikes a balance between speed and redundancy by stripping data across two drives but also writing "parity" data to a third. Parity data maintains a record of the differences between the blocks of data on the other drives, in turn permitting file restoration in the event of a drive failure. (A great explanation of parity and RAID 5 in detail can be found in this video. For mission-critical applications, "RAID 10" strips and mirrors data across four or more drives to provide the gold standard in performance and redundancy. You can find a more detailed explanation of RAID 0, 1, 5 and 10 on TheGeekStuff.com.

Many modern personal computer motherboards permit two SATA hard disk drives to set up in a RAID configuration. However, for users who do not require the extra speed provided by RAID 0, RAID 5 or RAID 10, there are relatively few benefits to be gained. Not least, it needs to be remembered that any hardware setup featuring more than one internal hard disk -- whether or not in a RAID configuration -- at best provides marginal improvements in data security and integrity. This is simply because it provides no more tolerance to the theft of the base unit, nor to power surges or computer power supply failures (which can simply fry two or more hard drives at once rather than one). A summary of RAID can also be found in my Explaining RAID video.

Logical unit number

In computer storage, a logical unit number, or LUN, is a number used to identify a logical unit, which is a device addressed by the SCSI protocol or protocols which encapsulate SCSI, such as Fibre Channel or iSCSI. A LUN may be used with any device which supports read/write operations, such as a tape drive, but is most often used to refer to a logical disk as created on a SAN. Though not technically correct, the term "LUN" is often also used to refer to the logical disk itself.
There is no 1:1 relationship between physical disk drives and LUNs. When provisioning storage, the administrator uses management software to create LUNs. They can create, for example, more than one LUN from one physical drive, which would then appear as two or more discrete drives to the user. Or they may create a number of LUNs that span several separate disks that form a RAID array; but, again, these will appear as discrete drives to users.
LUNs can be shared between several servers; for example, between an active server and a failover server. But problems can arise if a number of servers access the same LUN at the same time. There needs to be a method of ensuring data integrity because blocks are subject to change by the activities of those servers. For this, you need something like a clustered volume manager, clustered file system, clustered application or a network file system using NFS or CIFS.
In a SAN fabric, LUN storage is essential to the configuration of the environment and its performance. A LUN is a unique identifier given to separate devices, or logical units, so they can be accessed by a SCSI, iSCSI or Fibre Channel protocol. LUNs are key to disk array configuration because disks are typically defined in sets of RAID groups to protect against failure; however, those RAID groups can't be presented to the host. By assigning LUNs, all or a portion of a RAID group's capacity can be presented to the host as individual, mountable volumes.
From the computer perspective, SCSI LUN is only a part of the full SCSI address. The full device's address is made from the:
In the Unix family of operating systems, these IDs are often combined into a single "name". For example, /dev/dsk/c1t2d3s4 would refer to controller 1, target 2, disk 3, slice 4. Presently SolarisHP-UXNCR, and others continue to use "cXtXdXsX" nomenclature, while AIX has abandoned it in favor of more familiar names.


cXtXdXsX nomenclature in Unix
  • t-part: target ID identifying the SCSI target on that bus,
  • d-part: disk ID identifying a LUN on that target,
  • s-part: slice ID identifying a specific slice on that disk.

The cable and the host adapter form the SCSI bus, and this operates independently of the rest of the computer. Each of the eight devices is given a unique address by the SCSI BIOS, ranging from 0 to 7 for an 8-bit bus or 0 to 15 for a 16-bit bus. Devices that request I/O processes are called initiators.Targets are devices that perform operations requested by initiators. 

References:

Saturday, 29 March 2014

Wiping the dust off Motorola Defy

I have an old Motorola Defy, since I bought a new android phone I thought it would be a nice idea to have the other phone also to be an android rather than an old Nokia. But my idea was to sync my contacts if I eventually loose my phone. However Android 2.2 is kind of getting old. So this is what I did.

Charge your phone. Here I will make a note that I won't take any responsibility for your actions, but also if you mess up read down below - I did too once, don't worry.

Factory Data Reset

1. From the screen of your phone, open the Applications Tray. (circle located in the middle at the right of the display screen) Or press the Menu button on the left from your screen.
2. Tap Settings
3. Scroll down and tap Privacy
4. Tap Factory Data Reset
5. Tap Reset Phone

First get everything. Here are the downloads:
Start by dowloading a stable release of the ROM:
Gapps:
Here for other versions of CM in future - choose the appropriate for your release: http://wiki.cyanogenmod.org/w/Google_Apps

Transfer all files to the sdcard and extract the bootloader archive to a directory named bootloader on the sdcard.

Disconnect from the usb.
Install the applications Frama root and Terminal

Enable these:
Settings/Applications>Unknow Sources
Settings/Applications/Develpment>Development

Root:
Gain Root with Framaroot
Install it from the file manager
Use Framaroot for one click root. (Select Gimli.)

Use Terminal to install the bootmenu:
su
cp -R /sdcard/bootmenu /data/bootmenu
cd /data/bootmenu
chmod 755 *
./install.sh
rm -R /data/bootmenu
reboot

Time to flash:

1.  Next, navigate through the recovery mode menu and select “wipe data factory reset”, “wipe cache partition” and “wipe dalvick cache”. Very important this is why I messed up the first time.
2.  The system will now be wiped out, so that’s why the backup has been recommended before.
3.  Up next, return to recovery main menu: select “+++go back+++”.
4.  And choose “install zip from SD card” and “choose zip from SD card”. <- choose the ROM
5.  Pick the update file and flash the same.
6.  Optional: repeat this operation for applying the Google Apps package.
7. When the installation process ends, from recovery select “+++go back+++” and “reboot system now”.
8.  You are done.

Disable these:
Settings/Applications>Unknown Sources
Settings/Applications/Develpment>Development

In case of a mess (well it happens don't pannic ...):
Rename the rom to some short name - the RSD doesn't like long names.
Don't plug the device in the usb before you boot! (Power+volume up) 


References:

Thursday, 27 March 2014

Short reference - Linux commands, files, configurations etc

I haven't updated my command cheat sheat quite a while, but well today I found some time. Have fun hope it helps - Yuriy.

Short reference - Linux
(Commands, Directories, Configuration files/Scripts) 

Linux command
Description
example
start example where example is a
program found in the $PATH
./example
start example (allocated in current dir.)
adduser
add a user
rmuser
remove a user
passwd
change password
cd x or cd /x
go to directory x
cd .. or cd ../ or cd /..
go back one directory
pwd
Show current directory
ls or dir
list a directory
ls -al
Formated listing with hidden files
date
Show current date/time
cal
Show this months callender
uptime
Show uptime
w
Display who is online
whoami
Who are you logged in as
uname -a
Show kernel config
df
Show disk usage
du
Show directory space usage
du -sh
Human readable size in gigabyte
free
Show memory and swap usage
whereis app
Show possible locations of an app
which app
Show which app will be run by default
cat
display a textfile on the terminal

Examples:
cat /proc/meminfo - more memory info
cat /proc/pci - list pci devices, interrupts, io
cat /proc/cpuinfo - cpu info
cat /proc/version - kernel and gcc version info
cat /proc/ioports – input/output range of devices
cat /proc/devices - Displays various character and block devices that are currently configured.
cat /proc/filesystems - Lists all file system types currently supported by the kernel.
cat /proc/mdstat - Contains current information on multiple-disk or RAID configurations on the system, if they exist.
cat /proc/mounts - Lists all mounts currently used by the system.
cat /proc/partitions - Contains partition block allocation information.
cat > file
place standart input into a file
touch file
create or update file
more file
Displays the file - paging through text one screen at a time
less file
Displays the top of the file you can use the arrow keys and the Page-Up and Page-Down keys to move through the file
head file
first 10 lines of a file.
tail file
last 10 lines of a file
tail -f file
output contents of a file as it grows
mv x y
move or rename file x to file y
cp x y
copy file x to file y
rm -f x
force remove a file
rm -rf dir
rm -rf /old_stuff - correct
rm -rf / old_stuff – common mistake leaving space between / and the directory name. This will wipe your system
mkdir x
make a directory
mkdir x
make a directory
rmdir x
remove a directory
rm -r x
remove directory x and all its underlying files
man x
get a manual page about command x
less x or more x
view text file x
echo
print something on the screen
mainly used in scripts
mc
similar to Norton Commander in DOS
mount
to make a device usable to the system (CD-ROM,other HDD)
umount
remove the device from being usable to the system
halt
to shutdown. It will also power off if APM or ACPI is being used
reboot or ([ctrl] + [alt] + [del])
to reboot the system
chmod <permissions> <file>
change file permissions
chown <owner:group> <file>
change file ownership
ls -l x
perform a detailed directory listing for file or directory x
ln -s file link
make a symbolic link to "file" called "link"
top
view memory status (q to quit)
ps
to view all running processes
ps aux
ps with mored detail
kill x
to terminate process x (x is PID from ps)
kill pid
kill processes with pid 'pid'
killall -KILL pgm
to kill the program called pgm
killall -HUP pgm
to force program pgm to restart - used in case configuration is changed
killall proc
Kill all processes named proc
bg
List stoped/background jobs, resume stopped job in the background
fg
Bring most recent jobs to foreground
fg n
Brings job m to foreground
lilo
to write boot information changes to disk
ldd
shows the needed shared libraries
ldconfig
updates shared libraries
lsmod
shows loaded modules
Searching
Description
find x -name y -print
find file y, begin the search from within directory x and print the results to the screen
grep pattern files
Search for pattern in files
grep -r pattern dir
Search recursively for pattern in dir
command | grep pattern
Search recursively for pattern in dir
locate file
Find all instances of file
Networking
Description
whois domain
Get whois for domain
dig domain
Get DNS for domain
dig -x host
Reverse lookup host
wget -c file
Continue stopped download
wget -r url
Recursively download files from url
ssh user@host
Connect to host as a user
ssh -p port user@host
Connect using port p
ssh -D port user@host
Connect and use bind port
sshfs name@server:/path/folder /path/mount
Mount filesystem over ssh
ssh -t reachable_host ssh unreachable_host
Ssh trough host in the middle
python -m SimpleHTTPServer
Server current directory at http://$HOSTNAME:8000




Linux directories
Directory function
/
the root of the directory tree
/dev
the device pointer directory all devices have an entry here
/bin
main CLI executables
/sbin
main CLI system executables
/boot
location of the kernel and boot information required to boot the system
/etc
system configuration files
/root
root user home directory
/home
normal user base directory
/lib
system libraries and kernel modules
/usr
main user binary,documentation, library location eg. XFree86 and other X apps are here
/opt
extra binary/documentation,library location - eg.KDE
/var
system log files,spoolers,etc
/tmp
temporary folder for running applications that need temp files
/proc
includes files with information about the system
/mnt
filesystem mount point for devices (cdrom, usb etc.)


Linux configuration/script file
Description
/etc/lilo.conf
boot loader configuration file – you must run "lilo" after any changes
/etc/modprobe.conf
kernel module loader configuration used by the modprobe command
/etc/inittab
system runlevel configuration file - tells init what to do for each runlevel
/etc/fstab
filesystem mounting configuration file - used by the mount command
/etc/ld.so.conf
library configuration file - used by the ldconfig command to find libraries
/etc/inetd.conf
tcp wrapper configuration - used to start ftp and other network daemons
/etc/hosts
local host resolution file - manually enter hosts you want to connect to if they are not on DNS
/etc/HOSTNAME
system hostname file
/etc/profile
system defaults for bash login default paths,colours,etc
/etc/rc.d/rc.0
system halt script - run when you issue halt
/etc/rc.d/rc.6
system reboot script - run when you issue reboot or [ctrl] + [alt] + [del]
/etc/rc.d/rc.K
runlevel 1 or system administration mode
/etc/rc.d/rc.S
main system boot script - run when you boot up
/etc/rc.d/rc.M
main local startup script - activates other services you have chosen
/etc/rc.d/rc.sysvinit
SysV Init script starter
/etc/rc.d/rc.inet1
network startup script
/etc/rc.d/rc.modules
system module loader - run at startup
/etc/rc.d/rc.hotplug
hotplug starter script - for USB and PCI hotplug devices
/etc/rc.d/rc.pcmcia
pcmcia service script - for use with laptops
/etc/rc.d/rc.local
local startup script for user space programs
/etc/X11/xorg.conf
Xorg configuration file




Directories/files
Permission
Description
/var/log
751
Directory containing all log files
/var/log/messages
644      
System messages
/etc/crontab
600
System-wide crontab file
/etc/syslog.conf
640
Syslog daemon configuration file
/etc/logrotate.conf
640
Controls rotating of system log files
/var/log/wtmp
660          
Who is logged in now. Use who to view
/var/log/lastlog
640                             
Who has logged in before. Use last to view
/eto/ftpusers
600      
List of users that cannot FTP
/etc/passwd
644      
List of the system’s user accounts
/etc/shadow
600
Contains encrypted account passwords
/etc/pam.d
750
PAM configuration files
/etc/hosts.allow
600
Access control file
/etc/hosts-deny
600
Access control file
/etc/lilo.conf
600      
Boot loader configuration file
/etc/securetty
600      
TTY interfaces that allow root logins
/etc/shutdown.allow
400
Users permitted to ctrl-alt-del
/etc/security     
700
System access security policy files
/etc/rc.d/init.d
750
Program start-up files on Red Hat systems
/etc/init.d         
750
Program start-up files on Debian systems
/etc/sysconfig
751
System and network config files on Red Hat
/etc/inetd.conf
600      
Internet SuperServer configuration file
/etc/cron.allow
400      
List of users permitted to use cron
/etc/cron.deny
400      
List of users denied access to cron
/etc/ssh
750      
Secure Shell configuration files
/etc/sysctl.conf
400
Contains kernel tunable options on recent Red Hat