tag:blogger.com,1999:blog-63536436958182711452024-03-19T00:33:36.586-07:00WhiteBox Developer(Big stuffs in small blog)Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.comBlogger26125tag:blogger.com,1999:blog-6353643695818271145.post-90964036573440570022021-06-27T20:49:00.002-07:002021-06-27T20:49:48.409-07:00Hive: Repair Full Database<p>Hive provides a command to repair individual tables but lacks a command to repair full database at once</p>
<p>Following script can be used to repair full database</p>
<pre><code><span class="hljs-meta">#!/bin/bash</span>
<span class="hljs-keyword">if</span> [ <span class="hljs-variable">$#</span> <span class="hljs-_">-lt</span> 1 ]
<span class="hljs-keyword">then</span>
<span class="hljs-built_in">echo</span> <span class="hljs-string">"Error: Please supply database name to be repaired!"</span>
<span class="hljs-built_in">exit</span> 1
<span class="hljs-keyword">fi</span>
DB=<span class="hljs-variable">$1</span>
TABLES=<span class="hljs-string">"tables.hql"</span>
<span class="hljs-built_in">echo</span> <span class="hljs-string">"--"</span> &gt; <span class="hljs-variable">$TABLES</span>
<span class="hljs-keyword">for</span> TABLE <span class="hljs-keyword">in</span> `hive --database <span class="hljs-variable">$DB</span> -S <span class="hljs-_">-e</span> <span class="hljs-string">'show tables;'</span>`
<span class="hljs-keyword">do</span>
<span class="hljs-built_in">echo</span> <span class="hljs-string">"msck repair table <span class="hljs-variable">$TABLE</span>;"</span> &gt;&gt; <span class="hljs-variable">$TABLES</span>
<span class="hljs-keyword">done</span>
<span class="hljs-built_in">echo</span> <span class="hljs-string">"Repairing tables..."</span>
cat <span class="hljs-variable">$TABLES</span>
hive --database <span class="hljs-variable">$DB</span> <span class="hljs-_">-f</span> <span class="hljs-variable">$TABLES</span>
<span class="hljs-built_in">echo</span> <span class="hljs-string">"Done!"</span>
</code></pre>Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com1tag:blogger.com,1999:blog-6353643695818271145.post-58451070369316655162017-07-21T04:33:00.002-07:002017-07-24T20:59:54.703-07:00Java/JDK 9 Features at a Glance<pre> <div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-z2HMIw1xoag/WXHk1MI1pxI/AAAAAAAAAHg/6Bnd60vUOaAqjo0BAfdpv7zJakPpmWjawCLcBGAs/s1600/java9-features.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="388" data-original-width="750" height="165" src="https://1.bp.blogspot.com/-z2HMIw1xoag/WXHk1MI1pxI/AAAAAAAAAHg/6Bnd60vUOaAqjo0BAfdpv7zJakPpmWjawCLcBGAs/s320/java9-features.png" width="320" /></a></div>
</pre>
Java-9's full-fledged version is yet to be released but here I am going to discuss about its main developer features
with reference to <a href="http://jdk.java.net/9/" target="_blank">Early Access Build</a> of JDK-9 and <a href="http://www.javamagazine.mozaicreader.com/" target="_blank">Java Magazine</a> that I got from Oracle in my email
last night around 11 PM NPT.<br />
<br />
I will be explaining each feature with code examples where applicable and also provide implementation demo available at Github <a href="https://github.com/mehikmat/java-stuffs/blob/master/java9_features/README.md" target="_blank">Repo</a>.<br />
<br />
Though the primary bone of JDK-9 is supposed to be JPMS( Java Platform Modular System), I am not going to mention it here as
it's yet to be finalized by Java community.<br />
<br />
NOTE: if you are IntelliJ IDEA user, FYI, IDEA version lower than 17 doesn't support Java 9.<br />
<br />
<u><b>Nine Hot features inside of JDK-9 for Java hard-cores</b></u><br />
<br />
<pre>1) Factory Methods for Collections
See example: <a href="https://github.com/mehikmat/java-stuffs/blob/master/java9_features/src/main/java/Java9Collections.java" target="_blank">java-stuffs/java9_features/src/main/java/Java9Collections.java</a>
2) Optional Class Enhancement
See example: <a href="https://github.com/mehikmat/java-stuffs/blob/master/java9_features/src/main/java/OptionalClass.java" target="_blank">java-stuffs/java9_features/src/main/java/OptionalClass.java</a>
3) Stream API Enhancement
4) New tool: Read-Eval-Print-Loop(REPL) - JShell
Try: $JAVA_HOME/bin/jshell
5) Concurrency Updates
Java 7 added fork/join, java 8 added parallel streams, and now here java 9 has added publish-subscribe model to streams
6) Milling Project Coin
Private methods can be added to interfaces single underscore is no longer valid as variable name, in Jdk-10 it will be used as lambda variable
7) Spin-Wait-Hints
New method onSpinWait() has been added to Thread, supports optimization in JVM and Processor
8)Variable Handles
9) Process API Updates
In addition to Process and ProcessBuilder, jdk9 introduces new interface ProcessHandle interface.
This new interface is used to make a UNIX-style pipe line of Process Builders.
TODO: JPMS
<b>Reference: Java 9 Magazine July/August issue.</b>
</pre>
Anonymoushttp://www.blogger.com/profile/16168202560966858605noreply@blogger.com8tag:blogger.com,1999:blog-6353643695818271145.post-56180288178176256322016-03-29T23:11:00.001-07:002016-03-29T23:25:28.446-07:00Cloud Computing: A Look on Telnet and SSH<div style="color:black;font-size: 12px">
<h2><a id="Telnet_Traditional_way_to_connect_Remote_Host_0"></a>Telnet: Traditional way to connect Remote Host</h2>
<p>Overview,</p>
<ul>
<li><code>Telnet</code> is an OSI <code>Application Layer</code> protocol used to provide a <code>bidirectional interactive text-oriented communication facility</code> over <code>network</code> using a virtual <code>terminal connection</code>.</li>
<li><code>Telnet</code> provids access to a <code>command-line interface</code>of an operating system on a remote host/router.</li>
<li><code>Telnet</code> uses TCP <code>port 23</code> by default.</li>
</ul>
<p>Generally,</p>
<ul>
<li>The term <code>telnet</code> is used to refer to the software that implements the client part of the protocol.</li>
<li>The term <code>telnetd</code> is used to refer to the software that implements the server part of the protocol.</li>
</ul>
<p>Installation,</p>
<ul>
<li>Execute <code>$> sudo apt-get install telnetd</code> in case of <code>Ubuntu</code> and <code>$> sudo yum install telnetd</code> in case of <code>Centos</code> to install telnet server on remote host.</li>
<li>For client, use <code>telnet</code> commnad , a built-in telnet client, from host you want telnet to server.</li>
</ul>
<p>Now you can establish remote connection using command,</p>
<ul>
<li><code>$> telnet remote_host/ip 23</code></li>
</ul>
<p>Then you will get a terminal session to remote host’s terminal where you can execute any comand for remote host.</p>
<p>Disadvantag,</p>
<ul>
<li>Accessing a remote terminal through telnet program is unsecure since everything that you send or receive over that telnet session is visible in plain text. So anyone who can “sniff” (eg. using tools like <code>Wireshark</code> or <code>tcpdump</code>) the connection in-between can see your stuffs.</li>
<li>For this reason we need a more sophisticated program than telnet to connect to a remote host, ie. SSH :) .</li>
<li>N/W administrator can use tcpdump for capturing packtes and Wireshark to view/analyse the packets. eg. <code>tcpdump port 23 -w dump.txt</code> and <code>wireshark dump.txt</code></li>
</ul>
<h2><a id="SSH_Modern_way_to_connect_Remote_Host_26"></a>SSH: Modern way to connect Remote Host</h2>
<p>Overview,</p>
<ul>
<li>SSH provides <code>encrypted session</code>, <code>authentication facilities</code>, <code>secure file transfer(SCP|sftp)</code>, <code>X session forwarding</code>, and <code>port forwarding</code> etc.</li>
<li><code>SSH</code> is an OSI <code>Application Layer</code> protocol used to provide a <code>bidirectional interactive text-oriented communication facility</code> over <code>network</code> using a virtual <code>terminal connection</code>.</li>
<li><code>SSH</code> provids access to a <code>command-line interface</code>of an operating system on a remote host/router.</li>
<li><code>SSH</code> uses TCP <code>port 22</code> by default.</li>
</ul>
<p>Generally,</p>
<ul>
<li>The term <code>ssh</code> is used to refer to the software that implements the client part of the protocol.</li>
<li>The term <code>sshd</code> is used to refer to the software that implements the server part of the protocol.</li>
</ul>
<p>Installation,</p>
<ul>
<li>Execute <code>$> sudo apt-get install openssh-client</code> in case of <code>Ubuntu</code> and <code>$> sudo yum install openssh-client</code> in case of <code>Centos</code> to install telnet server on remote host.</li>
<li>Execute <code>$> sudo apt-get install openssh-server</code> in case of <code>Ubuntu</code> and <code>$> sudo yum install openssh-server</code> in case of <code>Centos</code> to install telnet server on remote host.</li>
<li>Generate private and public key using <code>$> ssh-keygen -t rsa</code></li>
<li>Copy your public key to server using <code>$> ssh-copy-id user@remotehost</code></li>
<li>Change the permission using <code>$> chmod 600 .ssh/authorized_keys</code></li>
</ul>
<p>Now you can establish remote connection using command,</p>
<ul>
<li><code>$> ssh -i private_key user@remote_host</code>
Then you will get a terminal session to remote host’s terminal where you can execute any comand for remote host.</li>
</ul>
<p>More on SSH,</p>
<h2><a id="X11_Session_Forwarding_using_SSH_51"></a><code>X11 Session Forwarding</code> using SSH</h2>
<ul>
<li>Using this feature of SSH, it’s possible to transmit window and bitmap information over a network connection. So you can login to a remote desktop and run some X windows program like Firefox and the program will run on the remote computer, but will display its graphical output on your local computer.</li>
<li>For this, <code>$> ssh -X user@remotehost</code> where -X option is for X11 session forwarding.</li>
</ul>
<h2><a id="TCP_Port_Forwarding_56"></a>TCP Port Forwarding</h2>
<p>You can setup a port forward from your localhost to remotehost so that it will take connections to localhost port 3306 and forward them to the remote side port 3306.
This way MySQL server would allow you to run GUI MySQL programs on your local computer while using the database on your remotehost.</p>
<p>Here is the command to accomplish this:</p>
<pre><code><span class="hljs-variable">$></span> ssh -<span class="hljs-constant">L</span> <span class="hljs-number">3306</span><span class="hljs-symbol">:mysql</span>.<span class="hljs-symbol">remotehost:</span><span class="hljs-number">3306</span> user<span class="hljs-variable">@remotehost</span>
<span class="hljs-constant">The</span> -<span class="hljs-constant">L</span> (<span class="hljs-constant">Local</span> port) takes one argument <span class="hljs-keyword">of</span>
<local-port><span class="hljs-symbol">:<connect-to-host></span><span class="hljs-symbol">:<connect-to-port></span>
</code></pre>
<p>This is a kind of VPN connection.</p>
<h2><a id="Running_Shell_scriptcommands_on_Remote_Host_70"></a>Running <code>Shell script/commands on Remote Host</code></h2>
<p>Sometimes you don’t really want to run a shell like Bash on the host you are connecting to. Maybe you just want to run a command and exit. This is very simply accomplished by putting the command you wish to run at the end of your ssh connection command.</p>
<pre><code>$> ssh user<span class="hljs-property">@remotehost</span> <span class="hljs-string">"ls -l /"</span>
Or more conveniently,
<pre><code>$ ssh -i /path_to_private_key -o ConnectTimeout=<span class="hljs-number">10</span> -o ServerAliveInterval=<span class="hljs-number">60</span>
-o StrictHostKeyChecking=<span class="hljs-literal">no</span> -o UserKnownHostsFile=<span class="hljs-regexp">/dev/</span> <span class="hljs-literal">null</span>
-o StrictHostKeyChecking= <span class="hljs-literal">no</span>
user<span class="hljs-property">@remotehost</span> <span class="hljs-string">"your script or commands to be executed on remote host"</span>
</code></pre>
</span></code></pre>
<h2><a id="Screen_Tool_with_SSH_for_remote_connection_management_82"></a><code>Screen</code> Tool with SSH for remote connection management</h2>
<p>This tool can be your best friend if you are remote administrator or you work on
remote terminal for long time. This helps to maintain multiple remote session on single screen and can be resumed the session even after connection disruption.</p>
<p>Installation:<br>
Ubuntu: <code>$> sudo apt-get install screen</code> <br>CentOS: <code>$> sudo yum install screen</code></p>
<p>Commands,</p>
<p><code>$> screen</code> to open the screen<br>
<code>$> exit</code> to quit screen<br>
<code>$> screen -S name_of_screen</code> to open the named screen<br>
<code>$> screen -ls</code> to see the list of screens</p><br>
<p><code>Ctrl a" "d"</code> to detach from screen<br>
<code>$> screen -rx name_of_screen</code> to reattach the already created screen</p><br>
<p><code>Ctrl a" "c"</code> to create a tab inside the screen<br>
<p><code>"Ctrl-a" "n"</code> to switch between tabs of same screen</p>
</div>Anonymoushttp://www.blogger.com/profile/16168202560966858605noreply@blogger.com29tag:blogger.com,1999:blog-6353643695818271145.post-86671288216807661332016-02-29T23:53:00.001-08:002016-02-29T23:58:08.732-08:00A Look on "Too many open files" Exception<div style="color:black;font-size: 12px">
<ul>
<li>This exception occurs when you are opening a file (or a pipe or a socket) but the
number of open file descriptors count already reached to a limit.
You can verify this by printing the stacktrace for the IOException.</li>
<li>Whenever a file/socket is opened, the operating system creates an entry to represent this file and stores its information. Each entry is represented by an integer value and this entry is termed as file descriptor.</li>
<li>Basically, Java class FileDescriptor provides a handle to the underlying machine-specific structure representing an open file, an open socket, or another source or sink of bytes.</li>
<li>The applications should not create FileDescriptor objects, they are mainly used in creation of FileInputStream orFileOutputStream objects to contain it.</li>
</ul>
<p><strong><strong>Check the open files of a process</strong></strong></p>
<pre><code><span class="hljs-preprocessor"># lsof -p <span class="hljs-number">12390</span> | wc -l</span>
</code></pre>
<p><strong><strong>Tuning file descriptor limits on Linux</strong></strong></p>
<ul>
<li>Linux limits the number of file descriptors that any one process may open.</li>
<li>The default limits are 1024 open files per process.</li>
<li>The command <code>ulimit -aS</code> displays the current limit, and <code>ulimit -aH</code>
displays the hard limit (above which the limit cannot be increased without tuning<br>
kernel parameters in /proc).</li>
<li>The file descriptor limit can be increased using the following procedure:</li>
</ul>
<pre><code>Edit /etc/security/limits.conf <span class="hljs-operator">and</span> <span class="hljs-built_in">add</span> <span class="hljs-operator">the</span> <span class="hljs-keyword">lines</span>:
* soft nofile <span class="hljs-number">1024</span>
* hard nofile <span class="hljs-number">65535</span>
This will increase <span class="hljs-operator">the</span> limits <span class="hljs-keyword">for</span> all users <span class="hljs-operator">of</span> <span class="hljs-operator">the</span> machine, <span class="hljs-keyword">if</span> you want <span class="hljs-built_in">to</span> <span class="hljs-built_in">do</span> this <span class="hljs-keyword">for</span> <span class="hljs-operator">a</span> specific user <span class="hljs-built_in">replace</span> * <span class="hljs-operator">with</span> that username.
</code></pre>
<p><strong><strong>Find open files limit per process (may be different for different processes)</strong></strong></p>
<p><code># ulimit -n</code></p>
<p><strong><strong>Count all opened files by all process</strong></strong></p>
<p><code># cat /proc/sys/fs/file-nr</code></p>
<p><strong><strong>Get maximum open files count allowed</strong></strong></p>
<p><code># cat /proc/sys/fs/file-max</code></p>
<p><strong><strong>Open file limit applied to a specific process while running</strong></strong></p>
<p><code># cat /proc/<pid>/limits</code></p>
<p><strong><strong><strong><strong><strong>Testing code for this is available:</strong></strong></strong></strong></strong>
<a href="https://github.com/mehikmat/java-stuffs/tree/master/ExceptionsTest">ExceptionsTest</a></p>
</div>Anonymoushttp://www.blogger.com/profile/16168202560966858605noreply@blogger.com5tag:blogger.com,1999:blog-6353643695818271145.post-90637875218432244642015-12-03T06:20:00.000-08:002015-12-03T06:21:11.833-08:00Session and Cookies<div style="color:black;font-size: 12px">
<h2><a id="What_is_web_Session_0"></a>What is web Session?</h2>
<ul>
<li>An abstract concept to represent a series of HTTP requests and responses
between a specific web browser and server. HTTP doesn’t support the
notion of a session.</li>
<li>A session’s data is stored on the server (only 1 session per client)</li>
<li>Sessions are often built on top of cookies.</li>
<li>The only data the client stores is a cookie holding a unique session ID</li>
<li>And on each page request, the client sends its session ID cookie, and the
server uses this to find and retrieve the client’s session data</li>
</ul>
<h2><a id="What_is_Cookie_11"></a>What is Cookie?</h2>
<ul>
<li>A cookie is data stored on the client</li>
<li><strong>Session Cookie:</strong> the default type; a temporary cookie that is stored only in the browser’s memory when the browser is closed, temporary cookies will be erased
can not be used for tracking long-term information safer, because no programs other than the browser can access them</li>
<li><strong>persistent cookie:</strong> one that is stored in a file on the browser’s computer
can track long-term information potentially less secure, because users (or programs they run) can open cookie files, see/change the cookie values, etc.</li>
</ul>
<h2><a id="ClientServer_communication_19"></a>Client/Server communication</h2>
<ul>
<li>Client’s browser makes an initial request to the server</li>
<li>Server notes client’s IP address/browser, stores some local session data, and sends a session ID back to client</li>
<li>client sends that same session ID back to server on future requests</li>
<li>server uses session ID to retrieve the data for the client’s session later, like a ticket given at a coat-check room</li>
</ul>
<h2><a id="Where_is_stored_session_data_27"></a>Where is stored session data?</h2>
<ul>
<li>On the client, the session ID is stored as a cookie with the name PHPSESSID/JSESSIONID/ASPSESSIONID</li>
<li>On the server, session data are stored as temporary files such as /tmp/sess_fcc17f071…</li>
<li>You can find out (or change) the folder where session data is saved using the session_save_path function</li>
<li>For very large applications, session data can be stored into a SQL database (or other destination) instead using the session_set_save_handler function</li>
</ul>
<h2><a id="What_is_session_time_out__35"></a>What is session time out ?</h2>
<ul>
<li>Because HTTP is stateless, it is hard for the server to know when a user
has finished a session</li>
<li>Ideally, user explicitly logs out, but many users don’t client deletes
session cookies when browser closes</li>
<li>Server automatically cleans up old sessions after a period of time
old session data consumes resources and may present a security risk
adjustable in PHP/JAVA/ASP server settings</li>
</ul>
<h2><a id="How_HTTP_is_stateless__45"></a>How HTTP is stateless ?</h2>
<ul>
<li>HTTP is a stateless protocol, which means that the connection between the browser and the server is lost once the transaction ends.</li>
<li>For each request, client makes new connection to server</li>
</ul>
</div>Anonymoushttp://www.blogger.com/profile/16168202560966858605noreply@blogger.com1tag:blogger.com,1999:blog-6353643695818271145.post-2653435186679472472015-11-05T01:16:00.001-08:002017-07-24T21:02:11.249-07:00An Outlook to Hadoop MapReduce Framework<html><head><title>Untitled Document.md</title><style></style></head><body id="preview">
<div style="color: black; font-size: 12px;">
<h2>
<a href="https://www.blogger.com/null" id="MapReduce_Operators_0"></a>MapReduce Operators</h2>
<ol>
<li>Mapper</li>
<li>Reducer</li>
<li>Combiner</li>
<li>Partitioner</li>
<li>Sorting [GropingComparator and KeyComparator]</li>
</ol>
<h2>
<a href="https://www.blogger.com/null" id="MapReduce_Framework_8"></a>MapReduce Framework</h2>
<ul>
<li>Shuffle: The process of moving map outputs to the reducers is known as shuffling</li>
<li>Sort: Each reduce task is responsible for reducing the values associated with several intermediate keys.The set of intermediate keys on a single node is automatically sorted by Hadoop before they are presented to the Reducer.</li>
<li>Speculative Execution: As most of the tasks in a job are coming to a close(nearly 95%),the Hadoop platform will schedule redundant copies of the remaining tasks across
several nodes which do not have other work to perform. This process is known as
speculative execution. When tasks complete, they announce this fact to the JobTracker.
Whichever copy of a task finishes first becomes the definitive copy. If other copies were executing speculatively, Hadoop tells the TaskTrackers to abandon the tasks and discard their outputs. By default it is true.</li>
</ul>
<h2>
<a href="https://www.blogger.com/null" id="Common_MapReduce_Operations_17"></a>Common MapReduce Operations</h2>
<ol>
<li>Filtering or Grepping</li>
<li>Parsing, Conversion</li>
<li>Counting, Summing</li>
<li>Binning, Collating</li>
<li>Distributed Tasks</li>
<li>Simple Total Sorting</li>
<li>Chained Jobs</li>
</ol>
<h2>
<a href="https://www.blogger.com/null" id="Advanced_MapReduce_Operations_27"></a>Advanced MapReduce Operations</h2>
<ol>
<li>GroupBy</li>
<li>Distinct</li>
<li>Secondary Sort</li>
<li>CoGroup/Join</li>
<li>Distributed Total Sort</li>
<li>Distributed Cache</li>
<li>Reading from HDFS programmetically</li>
</ol>
<h2>
<a href="https://www.blogger.com/null" id="Very_Advanced_Operations_37"></a>Very Advanced Operations</h2>
<ol>
<li>Classification</li>
<li>Clustering</li>
<li>Regression</li>
<li>Dimension Reduction</li>
<li>Evolutionary Algorithms</li>
</ol>
<h2>
<a href="https://www.blogger.com/null" id="Combiner_45"></a>Combiner</h2>
The best part of all is that we do not need to write any additional code
to take advantage of this! If a reduce function is both commutative
and associative, then it can be used as a Combiner as well.<br />
<h2>
<a href="https://www.blogger.com/null" id="Partitioner_51"></a>Partitioner</h2>
The key and value are the intermediate key and value produced by the map function.
The numReduceTasks is the number of reducers used in the MapReduce program
and it is specified in the driver program. It is possible to have empty partitions with no data (when no of partition is less then no of reducer). We do the assigned partition number modulo numReduceTasks to avoid illegal partitions if the system has a lesser number of possible reducers than the assigned partition number.
The partitioning phase takes place after the map/combine phase and before the reduce phase. The number of partitions is equal to the number of reducers. The data gets partitioned across the reducers according to the partitioning function<br />
<h2>
<a href="https://www.blogger.com/null" id="Sequence_File_58"></a>Sequence File</h2>
SequenceFiles are flat files consisting of binary key/value pairs.
SequenceFile provides SequenceFile.Writer, SequenceFile.Reader and
SequenceFile.Sorter classes for writing, reading and sorting respectively.<br />
<h2>
<a href="https://www.blogger.com/null" id="Counting_and_Summing_64"></a>Counting and Summing</h2>
For instance, there is a log file where each record contains a response time and
it is required to calculate an average response time.
Applications: Log Analysis, Data Querying<br />
<h2>
<a href="https://www.blogger.com/null" id="Binning_and_Collating_70"></a>Binning and Collating</h2>
IN>> word : list of page numbers containing word
Applications: Inverted Indexes, ETL,Max,Min<br />
<h2>
<a href="https://www.blogger.com/null" id="Filtering_or_Grepping_75"></a>Filtering or Grepping</h2>
Applications: Log Analysis, Data Querying, ETL, Data Validation<br />
<h2>
<a href="https://www.blogger.com/null" id="Distributed_Task_79"></a>Distributed Task</h2>
There is a large computational problem that can be divided into multiple parts
and results from all parts can be combined together to obtain a final result.
Applications: Physical and Engineering Simulations, Numerical Analysis, Performance Testing<br />
<h2>
<a href="https://www.blogger.com/null" id="Chaining_jobs_85"></a>Chaining jobs</h2>
<ul>
<li>
Method 1:
First create the JobConf object “job1” for the first job and set all the parameters with “input” as inputdirectory and “temp” as output directory. Execute this job: JobClient.run(job1). Immediately below it, create the JobConf object “job2” for the second job and set all the parameters with “temp” as inputdirectory and “output” as output directory. Finally execute second job: JobClient.run(job2).<br />
</li>
<li>
Method 2:
Create two JobConf objects and set all the parameters in them just like (1) except that you don’t use JobClient.run. Then create two Job objects with jobconfs as parameters:<br />
</li>
</ul>
Using the jobControl object, you specify the job dependencies and then run the jobs:<br />
<pre><code> Job job1=new Job(jobconf1)<span class="hljs-comment">; Job job2=new Job(jobconf2);</span>
JobControl jbcntrl=new JobControl("jbcntrl")<span class="hljs-comment">;</span>
jbcntrl.addJob(job1)<span class="hljs-comment">;</span>
jbcntrl.addJob(job2)<span class="hljs-comment">;</span>
job2.addDependingJob(job1)<span class="hljs-comment">;</span>
jbcntrl.run()<span class="hljs-comment">;</span>
</code></pre>
<h2>
<a href="https://www.blogger.com/null" id="Distinct_Values_Unique_Items_Counting_103"></a>Distinct Values (Unique Items Counting)</h2>
More to come here…<br />
</div>
</body></html>Anonymoushttp://www.blogger.com/profile/16168202560966858605noreply@blogger.com1tag:blogger.com,1999:blog-6353643695818271145.post-18988389723393982932015-03-20T05:55:00.002-07:002015-03-20T06:13:30.462-07:00Things to Consider while Indexing Large volume of Data to ElasticSearch<div style="color:black;font-size: 12px">
<p>Indexing large volume of data usually more than a TB from hadoop to <br/> elasticsearch may not work as your expectation,and to get the job <br/>
done you may need to make specific changes in both sides:</p>
<ul>
<li>ElasticSearch cluster side</li>
<li>Hadoop Cluster side</li>
</ul>
<h3 id="for-elasticsearch">For ElasticSearch</h3>
<ul>
<li>try max_open_file_descriptors to at least 32k or 64k</li>
<li>
try setting no replicas at all and later change it back to origional
</li>
<li>
turn off refreshing and later chage it back to origional
</li>
<li>disable flushing and later change it back to origional</li>
<li>try indices.memory.index_buffer_size to higher value than default 10%
and later chage it back to origional</li>
<li>disable index warm up and later change it back to origional</li>
</ul>
<h3 id="for-hadoop">For Hadoop</h3>
<ul>
<li>set number of tasks to 10-20</li>
<li>try bulk request with proper request size</li>
<li>try node_client instead of transport client</li>
<li>
<p>try multiple parallel clients </p>
(In hadoop it happens usually since each task run in parallel
but keep in mind that nothing comes for free so
too many parallel client may give high load to elasticsearch cluster)
<li>
<p>set the query parameter ReplicationType to async</p>
</li>
</li>
</ul>
<b><u>Note</u></b>
<p>Again keep in mind that nothing comes for free and you may get poor search experience if
you only optimize your elasticsearch cluster for high indexing throughput.
So good option is to optimize cluster for maximum indexing throughput at the time of indexing and
after completing the indexing of data optimize it for search.</p>
</div>Anonymoushttp://www.blogger.com/profile/16168202560966858605noreply@blogger.com7tag:blogger.com,1999:blog-6353643695818271145.post-12918094096646678332015-01-14T21:16:00.001-08:002015-01-14T21:20:35.397-08:00Processing data from AWS S3 using Cascading on Apache Hadoop YARN<div style="color:black;font-size: 12px">
<h2 id="s3-native-filesystem-uri-scheme-s3n">S3 Native FileSystem (URI scheme: s3n)</h2>
<p>A native filesystem for reading and writing regular files on s3.
The advantage of this filesystem is that you can access files on s3
that were written with other tools.
Conversely, other tools can access files written using Hadoop.
The disadvantage is the 5GB limit on file size imposed by s3.
For this reason it is not suitable as a replacement for HDFS
(which has support for very large files).</p>
<h2 id="s3-block-filesystem-uri-scheme-s3">S3 Block FileSystem (URI scheme: s3)</h2>
<p>A block-based filesystem backed by s3.
Files are stored as blocks, just like they are in HDFS.
This permits efficient implementation of renames.
This filesystem requires you to dedicate a bucket for the filesystem - you should not use an existing bucket containing files, or write other files to the same bucket.
The files stored by this filesystem can be larger than 5GB, but they are not
interoperable with other s3 tools.</p>
<h2 id="note">Note</h2>
<p>Note that for input you must create bucket manually and upload files in that bucket,
for output you must create bucket and directory specified for output must not already exist.</p>
<h2 id="configuration-for-s3">Configuration for S3</h2>
<p>Add the following entry to <code>hdfs-site.xml</code></p>
<pre><code>$ vim HADOOP_HOME/etc/hadoop/hdfs-site.xml
<property>
<name>fs.s3.awsAccessKeyId</name>
<value>AWS-ACCESS-KEY</value>
</property>
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>AWS-SECRET-KEY</value>
</property>
</code></pre>
<p>OR</p>
<p><code>You can supply these properties to FlowConnector as done in example below</code></p>
<p>OR</p>
<pre><code> s3n://ID:SECRET@BUCKET (use as url)
</code></pre>
<h2 id="example">Example:</h2>
<pre><code> $> HADOOP_HOME/bin/hadoop distcp hdfs://ip:9001/user/nutch/file.csv
s3://ACCESSKEY:SECRETKEY@bucket_name/
</code></pre>
<br/>
<h3 id="what-are-access-key-and-secret-key">What are Access Key and Secret Key</h3>
<h2 id="aws-access-key">AWS access key</h2>
<p>This is actually a username .
It is alphanumeric text string that uniquely identifies the
user who owns the account.
No two accounts can have the same AWS Access Key.</p>
<h2 id="aws-secret-key">AWS Secret key</h2>
<p>This key plays the role of a password .
It's called secret because it is assumed to be known by the owner only that's why,when you type it in the given box, its displayed as asterisk
or dots.
A Password with Access Key forms a secure information set that confirms the user's identity.
You are advised to keep your Secret Key in a safe place.</p>
<h2 id="disadvantages">Disadvantages</h2>
<p>Note that by using S3 as an input you lose the data locality optimization,
which may be significant.</p>
<h2 id="general-practise">General Practise</h2>
<p>The general best practise is to copy in data using distcp at the start of a workflow, then copy it out at the end, using the transient HDFS in between.</p>
<br/>
<h3>Cascading example app</h3>
<pre><code>public class Main {
public void run(String[] args) {
Properties properties = new Properties();
String accessKey = args[0];
String secretKey = args[1];
// better put these keys to hadoop xml file
// for block file system
properties.setProperty("fs.s3.awsAccessKeyId", accessKey);
properties.setProperty("fs.s3.awsSecretAccessKey", secretKey);
// for s3 native file system
// properties.setProperty("fs.s3n.awsAccessKeyId", accessKey);
// properties.setProperty("fs.s3n.awsSecretAccessKey", secretKey);
// properties.setProperty("fs.defaultFS", "hdfs://localhost:8020/");
// properties.setProperty("fs.permissions.umask-mode", "007");
AppProps.setApplicationJarClass(properties, Main.class);
HadoopFlowConnector flowConnector = new HadoopFlowConnector(
properties );
String input = "s3://my-bucket/my-log.csv";
Tap inTap = new Hfs( new TextDelimited( false, ";" ), input);
Pipe copyPipe = new Pipe( "copy" );
Tap outTap = new Hfs( new TextDelimited( false, ";" ),
"data/output");
FlowDef flowDef = FlowDef.flowDef()
.addSource( copyPipe, inTap )
.addTailSink( copyPipe, outTap );
flowConnector.connect( flowDef ).complete();
}
public static void main(String[] args) {
new Main().run(args);
}
}
</code></pre>
</div>Anonymoushttp://www.blogger.com/profile/16168202560966858605noreply@blogger.com14tag:blogger.com,1999:blog-6353643695818271145.post-25753018524239702672015-01-02T01:01:00.001-08:002015-01-02T01:09:10.275-08:00Setting up Driven and Driven Plugin for Cascading Application<div style="color:black;font-size: 12px">
<h2 id="what-is-driven">What is Driven</h2>
<p>Driven is a client-server based application for Performance Management
and Operational Visualization for Cascading Based Applications.</p>
<h2 id="driven-consists-of-two-components">Driven consists of two components</h2>
<ul>
<li><em>Driven Server:</em> is a web-application that is hosted in web server.</li>
<li><em>Driven Client (Driven Plugin):</em> is the java application which is configured with Cascading Applications to be monitored.</li>
</ul>
<h2 id="note">NOTE:</h2>
<ul>
<li>You do not need to make any changes to existing Cascading applications to integrate with the Driven system.</li>
<li>Driven supports Cascading based all applications including
Scalding, Cascalog, Lingual, Pattern), Apache Hive, and native MapReduce.</li>
</ul>
<h2 id="how-to-setup-driven-for-cascading-application">How to setup Driven for cascading application</h2>
<p>Driven can be set up in two ways depending on your requirement.</p>
<h4 id="driven-for-enterprise">Driven For Enterprise</h4>
<p>In this way one can deploy driven server anywhere on their own environment.</p>
<p><em>Core features available</em></p>
<pre><code>- Application Timeline
- Application Runtime Visibility
- Team Collaboration & Sharing
- Complete History Retention
- Rich Search
- Powered by a meta-data repo
</code></pre>
<p><em>Enterprise features available</em></p>
<pre><code>_ Securely host in your own environment
_ Integration with Third-Party Monitoring Tools
_ Command-line Interface
</code></pre>
<p>if you want to test driven features (Don't have money to buy license)</p>
<ul>
<li>
<p>Sing up to http://cascading.io/trial/ for trial and you will be provided
with a 30-day trial license key and can download Driven to host
in your own environment.</p>
</li>
<li>
<p>Unzip the driven-tomcat package and start</p>
</li>
<li>
<p>$> DRIVEN_HOME/bin/startup.sh</p>
</li>
</ul>
<p>After starting the server go to http://<hostname>:8080.
There you will be presented with a form to enter license information
needed to complete the Driven server installation.</p>
<p>Login with username:admin and password:admin (default)
and an API key will also be provided which should be used with drive plugin.</p>
<p>Customize driven server by changing default configuration located at
SERVER_HOME/conf/driven.properties</p>
<h4 id="driven-early-access">Driven Early Access</h4>
<p>In this way one cannot deploy driven server anywhere on their own
environment and need to use concurrent (driven.cascading.io) hosted driven server.</p>
<p>All the core features are available but Enterprise features are not available.</p>
<p>In this cas you need to login using your credentials and then an API key will be provided to use with driven plugin.</p>
<h2 id="set-up-driven-plugin">Set up driven plugin</h2>
<p>Settin up driven plugin is similar for above both ways.</p>
<p>Add the repository</p>
<pre><code><repository>
<id>conjars</id>
<url>http://conjars.org/repo</url>
</repository>
</code></pre>
<p>Add driven plugin dependency</p>
<pre><code><dependency>
<groupId>driven</groupId>
<artifactId>driven-plugin</artifactId>
<version>1.2-eap-4</version>
<!--if you want use concurrent hosted driven server
<classifier>io</classifier>-->
</dependency>
</code></pre>
<p><strong>If you are running your application on Hadoop</strong></p>
<ul>
<li>Create a file named cascading-service.properties and put the following property</li>
</ul>
<pre><code>cascading.management.document.service.apikey = YOUR_API_KEY
cascading.management.document.service.hosts = YOUR_DRIVEN_SERVER_URL
(Don't specify host property if are using hosted driven)
</code></pre>
<ul>
<li>Copy this file inside hadoop conf dir. That's all.</li>
</ul>
<p><strong>If you are running your application in Local Mode</strong></p>
<p>Export the following properties to your environment</p>
<pre><code>export DRIVEN_API_KEY=_YOUR_API_KEY_
export DRIVEN_SERVER_HOSTS=_YOUR_DRIVEN_SERVER_URL_
(don't export host property if you are using hosted driven)
</code></pre>
</div>Anonymoushttp://www.blogger.com/profile/16168202560966858605noreply@blogger.com0tag:blogger.com,1999:blog-6353643695818271145.post-58554871345341739952014-11-16T21:22:00.000-08:002014-11-16T21:25:47.330-08:00Setting up LZO compression in Hadoop/YARN<div style="color:black;font-size: 12px">
<p>Install liblzo</p>
<ul>
<li>On Redhat based systems
<code>sudo yum install liblzo-devel</code></li>
<li>On Debian based systems
<code>sudo apt-get install liblzo2-dev</code></li>
</ul>
<p>Clone the hadoop-lzo from github</p>
<pre><code>$ git clone https://github.com/twitter/hadoop-lzo.git
$ cd hadoop-lzo
$ mvn clean package
</code></pre><p>Place the hadoop-lzo-*.jar somewhere on your cluste classpath</p>
<pre><code>$ cp hadoop-lzo/target/hadoop-lzo-0.4.20-SNAPSHOT.jar /data/lib/
</code></pre><p>Place the native hadoop-lzo binaries in hadoop native directory</p>
<pre><code>$ cp hadoop-lzo/target/native/Linux-amd64-64/lib/* $HADOOP_HOME/lib/native/hadoop-lzo/
</code></pre><p>Add the following to your <code>core-site.xml</code>:</p>
<pre><code><property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec
</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
</code></pre><p>Add the following to your <code>mapred-site.xml</code>:</p>
<pre><code><property>
<name>mapred.child.env</name>
<value>JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native/hadoop-lzo/</value>
</property>
<property>
<name>mapred.map.output.compression.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
</code></pre><h3 id="references-">References:</h3>
<ul>
<li><a href="http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh5ig_yarn_cluster_deploy.html">http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh5ig_yarn_cluster_deploy.html</a></li>
<li><a href="http://raseshmori.wordpress.com/2012/10/14/install-hadoop-nextgen-yarn-multi-node-cluster/">http://raseshmori.wordpress.com/2012/10/14/install-hadoop-nextgen-yarn-multi-node-cluster/</a></li>
<li><a href="https://www.digitalocean.com/community/articles/how-to-install-hadoop-on-ubuntu-13-10">https://www.digitalocean.com/community/articles/how-to-install-hadoop-on-ubuntu-13-10</a></li>
<li><a href="http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh5ig_mapreduce_to_yarn_migrate.html">http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh5ig_mapreduce_to_yarn_migrate.html</a></li>
</ul>
</div>Anonymoushttp://www.blogger.com/profile/16168202560966858605noreply@blogger.com35tag:blogger.com,1999:blog-6353643695818271145.post-31073733062253344082014-11-16T21:16:00.000-08:002014-11-16T21:20:47.702-08:00Building/Compiling Hadoop Native Library<div style="color:black;">
<p>If you see WARN like </p>
<pre><code>WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using<br/>
builtin-java classes where applicable
</code></pre>
<br/>
<p>Then you have to re-compile hadoop native code manually and then put libhadoop-*.so in classpath</p>
<p>Prerequistes:</p>
<ul>
<li>$ sudo apt-get install build-essential</li>
<li>$ sudo apt-get install g++ autoconf automake </li>
</ul>
<p>And make sure that cmake is installed correctly.</p>
<pre><code>$ cd $HADOOP_HOME/src
$ mvn package -Pdist,native -Dskiptests -Dtar
You should see the newly-built library in:
$ hadoop-dist/target/hadoop-2.3.0-cdh5.1.0/lib/native
Put following lines in hadoop-env.sh file
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/"
export HADOOP_COMMON_LIB_NATIVE_DIR="path/to/native"
OR simply,
$ cp $HADOOP_HOME/src/hadoop-dist/target/hadoop-2.3.0-cdh5.0.1/lib/native/* $HADOOP_HOME/lib/native/
</code></pre>
</div>Anonymoushttp://www.blogger.com/profile/16168202560966858605noreply@blogger.com0tag:blogger.com,1999:blog-6353643695818271145.post-243261840823760732014-11-14T00:06:00.001-08:002015-03-30T21:50:30.493-07:00Installing CDH 5 Hadoop YARN on a Single Linux Node in Pseudo-distributed Mode<div STYLE="margin-top: -34px; margin-left: 3px; font-size: 12px;
color: black"><br />
<br />
<p>For development purpose, Apache Hadoop and CDH 5 components can be deployed on a single Linux node in pseudo-distributed mode. In pseudo-distributed mode, Hadoop processing is distributed over all of the cores/processors on a single machine. Hadoop writes all files to the Hadoop Distributed File System (HDFS), and all services and daemons communicate over local TCP sockets for inter-process communication.</p><h4 id="prerequisites">Prerequisites</h4><ul><li>Supported Operating Systems: RedhatEL,Ubuntu,Debian,CentOS,SLES,OracleLinux->64-bit</li>
<li>Supported JDK Versions: >= jdk-1.7.0<em>25</em></li>
<li>Supported Internet Protocol: CDH requires IPv4. IPv6 is not supported.</li>
<li>SSH configuration:SSH should be configured</li>
</ul><h3 id="step-1">STEP-1</h3><pre><code>Download CDH tarball from cloudera
$ wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.3.0-cdh5.0.1.tar.gz
$ cd /opt
$ tar -xzf ~/hadoop-2.3.0-cdh5.0.1.tar.gz
$ cd hadoop-2.3.0-cdh5.0.1
</code></pre><p><em>Edit config files</em></p><ul><li>core-site.xml</li>
<li>hdfs-site.xml</li>
<li>mapred-site.xml</li>
<li>yarn-site.xml</li>
<li>hadoop-env.sh</li>
<li>yarn-env.sh</li>
<li><p>mapred-env.sh</p><p>OR</p></li>
</ul><pre><code>$ git clone https://github.com/mehikmat/hadoop-install.git
$ cp -R hadoop-install/etc/hadoop/* $HADOOP_HOME/etc/hadoop/
</code></pre><h3 id="step-2">STEP-2</h3><p>Create dirs,user, and set Java Home for all users</p><pre><code>$ git clone https://github.com/mehikmat/hadoop-install.git
$ cd hadoop-install/users-and-dirs
Set Java for all users
$ ./java.sh
OPTIONAL:> #Create users(if you want use separate users)
$ ./users.sh # no need to create multiple users in single node
Create required directories
$ ./dirs.sh # edit HDFS_USER,YARN_UER,and MAPRED_USER variables in this file to point same user
</code></pre><p><em>Edit ~/.bashrc</em> [optionally for file of hdfs and yarn user]</p><pre><code>export HADOOP_HOME=/opt/hadoop-2.3.0-cdh5.0.1
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
</code></pre><h5 id="refresh-bash-profile-bash-">Refresh bash profile <code>$ bash</code></h5><h2 id="step-3">STEP-3</h2><p><em>Create HDFS dirs</em></p><pre><code>Create the history directory and set permissions and owner
$ sudo -u hdfs hdfs dfs -mkdir -p /user/log/history OR [hdfs dfs -mkdir -p /user/log/history]
$ sudo -u hdfs hdfs dfs -chmod -R 1777 /user/log/history OR [hdfs dfs -chmod -R 1777 /user/log/history]
$ sudo -u hdfs hdfs dfs -chown mapred:hadoop /user/log/history OR [hdfs dfs -chown mapred:hadoop /user/log/history]
$ sudo -u hdfs hadoop fs -mkdir /tmp OR [hadoop fs -mkdir /tmp]
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp OR [hadoop fs -chmod -R 1777 /tmp]
</code></pre><h2 id="step-4">STEP-4</h2><p>Format HDFS</p><pre><code>If you have created separate user for each daemon
$ sudo -u hdfs bin/hdfs namenode -format
else
$ bin/hdfs namenode -format
</code></pre><h2 id="step-5">STEP-5</h2><p><em>Start HDFS and YARN services</em></p><pre><code>If you have created separate user for each deamon
$ sudo -u hdfs sbin/start-dfs.sh
$ sudo -u yarn sbin/start-yarn.sh
else
$ sbin/start-dfs.sh
$ sbin/start-yarn.sh
</code></pre><h3 id="utilities-">Utilities:</h3><ul><li>$HADOOP_HOME/bin/hadoop :>>> For basic hadoop operations</li>
<li>$HADOOP_HOME/bin/yarn :>>> For YARN related operations</li>
<li>$HADOOP_HOME/bin/mapred :>>> For MapReduce realted operations</li>
<li>$HADOOP_HOME/bin/hdfs :>>> For HDFS related operations</li>
</ul><h3 id="demoen-utilities-">Demoen Utilities:</h3><ul><li>$HADOOP_HOME/sbin/start-yarn.sh;stop-yarn.sh</li>
<li>$HADOOP_HOME/sbin/start-dfs.sh;stop-dfs.sh </li>
<li>$HADOOP_HOME/sbin/start-all.sh;stop-all.sh </li>
<li>$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver</li>
</ul><h2 id="step-6">STEP-6</h2><p>Check installation using <code>jps</code></p><pre><code>$ jps
20803 NameNode
22056 JobHistoryServer
22124 WebAppProxyServer
7926 Main
21817 NodeManager
21560 ResourceManager
8018 RemoteMavenServer
21373 SecondaryNameNode
21049 DataNode
25651 ElasticSearch
28730 Jps
</code></pre><p>If these services are not up, check the logs in <code>logs</code> directory to identify the issue.</p><h3 id="web-interfaces">Web interfaces</h3><ul><li>NameNode:> <a href="http://master:50070/dfshealth.jsp">http://master:50070/dfshealth.jsp</a></li>
<li>ResourceManager:> <a href="http://master:8088/cluster">http://master:8088/cluster</a></li>
<li>JobHistoryServer:> <a href="http://master:19888/jobhistory">http://master:19888/jobhistory</a></li>
</ul><h4 id="what-is-on-what">What is on what</h4><pre><code>Master Node:
- NameNode
- ResousrceManager
- JobHistoryServer
Slave Node:
- NodeManager
- DataNode
- WebAppProxyServer
</pre></div>Anonymoushttp://www.blogger.com/profile/16168202560966858605noreply@blogger.com19tag:blogger.com,1999:blog-6353643695818271145.post-79569219118346894172014-08-18T22:59:00.001-07:002014-08-18T23:04:26.655-07:00Apache Lucene: Hello World Example<div dir="ltr" style="text-align: left;" trbidi="on"><p>Apache Lucen is a full text-search library for java which helps<br />
you add search capability to your application/website. <br />
Note that Lucene is specifically an API, not an application.<br />
<br />
Apache Lucene is a power full search library on which the<br />
widely used search engines like Apache Solr,ElasticSearch are based</p><p>You can add contents to Lucene from various sources,like a SQL/NoSQL database, a filesystem, or even from websites.</p><h1 id="some-terminologies">Some Terminologies</h1><h2 id="index">Index</h2><p>Lucene uses index called an inverted index, because it inverts<br />
a page-centric data structure (page->words) to a keyword-centric<br />
data structure (word->pages).</p><h2 id="document">Document</h2><p>An index consists of one or more Documents.<br />
Indexing involves adding Documents to an IndexWriter,<br />
and searching involves retrieving Documents from an index via an IndexSearcher. </p><h2 id="fields">Fields</h2><p>A Document consists of one or more Fields. A Field is simply a name-value pair.</p><h1 id="lets-get-started">Let's get started:</h1><p>For quickly getting started I am going to create an In-Memory Index.</p><h4 id="step-1">Step-1</h4><p>Create an empty Maven project using the following console command:</p><pre><code>$ mvn archetype:create -DgroupId=demo.lucene -DartifactId=lucene-demo
</code></pre><br />
<h4 id="step-2">Step-2</h4><p>Add Lucene dependencies to <code>pom.xml</code> file as defined below</p><pre><code><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>demo.lucene</groupId>
<artifactId>lucene-demo</artifactId>
<version>1.0</version>
<name>Lucene-HelloWorld</name>
<description>Hello World Lucene Search Example</description>
<dependencies>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-core</artifactId>
<version>2.4.1</version>
</dependency>
<dependency>
<groupId>lucene</groupId>
<artifactId>lucene</artifactId>
<version>1.4.3</version>
</dependency>
</dependencies>
</project>
</code></pre><br />
<h4 id="step-3">Step-3</h4><p>Create a Java file <code>HelloWorldLucene.java</code> class</p><pre><code>import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.StringField;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.queryparser.classic.ParseException;
import org.apache.lucene.queryparser.classic.QueryParser;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.TopScoreDocCollector;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.Version;
import java.io.IOException;
public class HelloLucene {
public static void main(String[] args) throws IOException, ParseException {
// Specify the analyzer for tokenizing text.
// The same analyzer should be used for indexing and searching
StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_40);
// Create the index
Directory index = new RAMDirectory();
IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_40, analyzer);
IndexWriter w = new IndexWriter(index, config);
addDoc(w, "Lucene in Action", "193398817");
addDoc(w, "Lucene for Dummies", "55320055Z");
addDoc(w, "Managing Gigabytes", "55063554A");
addDoc(w, "The Art of Computer Science", "9900333X");
w.close();
//Query
String querystr = args.length > 0 ? args[0] : "lucene";
// the "title" arg specifies the default field to use
// when no field is explicitly specified in the query.
Query q = new QueryParser(Version.LUCENE_40, "title", analyzer).parse(querystr);
// Search
int hitsPerPage = 10;
IndexReader reader = DirectoryReader.open(index);
IndexSearcher searcher = new IndexSearcher(reader);
TopScoreDocCollector collector = TopScoreDocCollector.create(hitsPerPage, true);
searcher.search(q, collector);
ScoreDoc[] hits = collector.topDocs().scoreDocs;
// Display results
System.out.println("Found " + hits.length + " hits.");
for(int i=0;i<hits.length;++i) {
int docId = hits[i].doc;
Document d = searcher.doc(docId);
System.out.println((i + 1) + ". " + d.get("isbn") + "\t" + d.get("title"));
}
// reader can only be closed when there
// is no need to access the documents any more.
reader.close();
}
private static void addDoc(IndexWriter w, String title, String isbn) throws IOException {
Document doc = new Document();
doc.add(new TextField("title", title, Field.Store.YES));
// Use a string field for isbn because we don't want it tokenized
doc.add(new StringField("isbn", isbn, Field.Store.YES));
w.addDocument(doc);
}
}
</code></pre><br />
<h4 id="step-4">Step-4</h4><p>Compile project</p><p><code>$ mvn clean package</code></p><h4 id="step-5">Step-5</h4><p>Run Project</p><p><code>$ java -jar <jar_name> <Main-Class_name> <query_string></code></p><p>In our case run command:</p><p><code>$ java -jar lucene-demo-1.0.jar HelloWorldLucen "lucene"</code></p></div>Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com17tag:blogger.com,1999:blog-6353643695818271145.post-16128431505375952332014-04-28T05:31:00.004-07:002014-04-28T05:34:20.263-07:00Introduction to Gradle<div dir="ltr" style="text-align: left;color:black;" trbidi="on"><h2>Gradle Introduction</h2><ul><li>Gradle is full-flaged build tool for <em>Java</em>, <em>Scala</em>, <em>Groovy</em> and more.</li>
<li>Gradle requires a JDK 1.5 or higher.</li>
<li>Gradle ships with its own Groovy library, therefore no Groovy needs to be installed.</li>
<li>Any existing Groovy installation is ignored by Gradle.</li>
</ul><h2>Gradle Installation</h2><p>Gradle releases come in three flavours:</p><p>Example:<br />
- gradle-1.11-all.zip (binaries, sources and documentation)<br />
- gradle-1.11-bin.zip (binaries only)<br />
- gradle-1.11-src.zip (sources only)</p><p>To download Gradle version 1.11</p><p><code>$ wget https://services.gradle.org/distributions/gradle-1.11-all.zip</code><br />
For running Gradle, add <code>GRADLE_HOME/bin </code> to your <code> PATH </code> environment variable.<br />
Usually, this is sufficient to run Gradle.</p><p><code>$ echo "export PATH=$PATH:$GRADLE_HOME/bin" >>~/.bashrc</code></p><p>To check if Gradle is properly installed just type</p><p><code>$ gradle -v</code></p><p>The output shows Gradle version and also local environment configuration (groovy and jvm version, etc.)</p><h2>Build Script Basics</h2><ul><li>Everything in Gradle sits on top of two basic concepts: <strong>projects</strong> and <strong>tasks</strong>.</li>
<li>Every Gradle build is made up of one or more projects.</li>
<li>What a project represents depends on what it is that you are doing with Gradle.<br />
For example, a project might represent a library JAR or a web application or else things.</li>
<li>Each project is made up of one or more tasks.</li>
<li>A task represents some atomic piece of work which a build performs.</li>
<li>This might be compiling some classes, creating a JAR, generating javadoc, or publishing some<br />
archives to a repository.</li>
</ul><p>You run a Gradle build using the <strong>gradle</strong> command.<br />
The gradle command looks for a file called <strong>build.gradle</strong> in the current directory.<br />
For example, the very simple build script may be like below</p><p><em>build.gradle</em></p><pre><code>task hello {
doLast {
println 'Hello world!'
}
}</code></pre><p>In a command-line shell, enter into the containing directory and<br />
execute the build script by running</p><pre><code>$ gradle -q hello
Hello world!</code></pre><p>You can declare dependencies between your tasks. </p><p><em>build.gradle</em></p><pre><code>task hello << {
println 'Hello world!'
}
task intro(dependsOn: hello) << {
println "I'm Gradle"
}</code></pre><h2>Gradle Plugins</h2><p>Gradle comes with number plugins which provide some pre-configured tasks.<br />
- Java Plugin<br />
- Groovy Plugin<br />
- Web Plugin<br />
- Jetty Plugin<br />
- and more....</p><h3>Java Plugin</h3><p>Gradle Java Plugin adds some pre-cofigured tasks to your build configuration script to compile,run unit tests and create jar etc.</p><p>To use the Java plugin, add the following to your build file:</p><p><em>build.gradle</em></p><pre><code>apply plugin: 'java'</code></pre><p>And this plugin adds following tasks</p><ul><li><strong>build</strong><br />
Gradle will compile and test your code, and create a JAR file containing your main classes <br />
and resources</li>
<li><strong>clean</strong><br />
Deletes the build directory, removing all built files.</li>
<li><strong>assemble</strong><br />
Compiles and jars your code, but does not run the unit tests. Other plugins add more<br />
artifacts to this task. For example, if you use the War<br />
plugin, this task will also build the WAR file for your project. </li>
<li><strong>check</strong><br />
Compiles and tests your code. Other plugins add more checks to this task. For example, if you use the Code-quality plugin, this task will also <br />
run Checkstyle against your source code. </li>
</ul><h2>External dependencies</h2><p>Usually, a Java project will have some dependencies on external JAR files.<br />
To reference these JAR files in the project, you need to tell Gradle where to find them.<br />
In Gradle, artifacts such as JAR files, are located in a repository.<br />
A repository can be used for fetching the dependencies of a project, or for publishing the artifacts of a project, or both.<br />
For example, we will use the public Maven repository:</p><p><em>build.gradle</em></p><pre><code>repositories {
mavenCentral()
}</code></pre><p>Now we will declare that our production classes have a compile-time dependency on commons collections,<br />
and that our test classes have a compile-time dependency on junit:</p><p><em>build.gradle</em></p><pre><code>dependencies {
compile group: 'commons-collections', name: 'commons-collections', version: '3.2'
testCompile group: 'junit', name: 'junit', version: '4.+'
}</code></pre><h2>Multi-project Java build</h2><p>For example following may be our multi-project layout</p><pre><code>multiproject/
api/
services/webservice/
shared/</code></pre><p>To define a multi-project build, you need to create a settings file and include following lines</p><p><em>settings.gradle</em></p><p><code>include "shared", "api", "services:webservice", "services:shared"</code></p><h2>Multi-project build - dependencies between projects</h2><pre><code>api/build.gradle
dependencies {
compile project(':shared')
}</code></pre><h2>Obtaining information about build configuration</h2><ul><li>Listing Project: <code>gradle -q projects</code></li>
<li>Listing Tasks: <code>gradle -q tasks</code></li>
<li>Showing task usage: <code>gradle -q help --task libs</code></li>
<li>Listing project dependencies: <code>gradle -q dependencies api:dependencies webapp:dependencies</code></li>
<li>Listing project properties: <code>gradle -q api:properties</code></li>
<li>Dry Run: <code>gradle -m clean compile</code></li>
</ul><h2>Gradle Graphical User Interface</h2><p><code>$ gradle --gui</code></p><h2>Gradle Build Language</h2><p>Gradle provides a domain specific language, or DSL, for describing builds.<br />
This build language is based on Groovy, with some additions to make it easier to describe a build.</p><h2>Build Init Plugin</h2><p>The Gradle Build Init plugin can be used to bootstrap the process of creating a new Gradle build.<br />
It supports creating brand new projects of different types as well as converting existing builds<br />
(e.g. An Apache Maven build) to be Gradle builds.<br />
The Build Init plugin is an automatically applied plugin,<br />
which means you do not need to apply it explicitly to your project. </p><p><strong>Build Init plugin - tasks</strong></p><ul><li><strong>init</strong> :Generates a Gradle project.</li>
<li><strong>wrapper</strong> :Generates Gradle wrapper files.</li>
</ul><p>The init supports different build setup types.<br />
The type is specified by supplying a <code>--type argument</code> value.</p><pre><code>$ gradle init --type java-library [for java project]
$ gradle init --type pom [to convert existing maven project to gradle]
$ gradle init --type scala-library [for scala project]
$ gradle init --type groovy-library [for groovy project]
$ gradle init --type basic [for fresh new project]</code></pre><p>If a <code>--type parameter</code> is not supplied, Gradle will attempt to infer the type from the environment.<br />
For example, it will infer a type value of "pom" if it finds a pom.xml to convert to a Gradle build.<br />
If the type could not be inferred, the type <code>"basic"</code> will be used.</p><br />
<br />
<br />
<b>References:</b><br />
<br />
http://www.gradle.org/docs/current/userguide<br />
</div>Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com4tag:blogger.com,1999:blog-6353643695818271145.post-88148053239502188922014-04-23T22:01:00.002-07:002014-04-23T22:05:10.139-07:00Creating Multiple Users in Hadoop Cluster<div dir="ltr" style="text-align: left;color:black;" trbidi="on"><h3>What happens if we share same user among multiple users?</h3><hr><p>If we share same hadoop user account among multiple users then we suffer from following problems <br />
- Difficult to trace the jobs and track the tasks/defects done by each user<br />
- Shared account arriase security issue.<br />
- If all are given the same user account, all users will have the same privilege and all can<br />
access everyone’s data, can modify it, can perform execution, can delete it also.</p><h3>What are the benefits of creating multiple users?</h3><hr><ul><li>The directories/files of one user cannot be modified by other user.</li>
<li>Other users cannot add new files to a user’s directory.</li>
<li>Other users cannot perform any tasks (mapreduce etc) on a user’s files.</li>
</ul><h2>Steps for setting up multiple User Accounts</h2><hr><h4>[1] Create a New User</h4><pre><code> Ubuntu: $ sudo adduser --ingroup <groupname> <username>
Redhat: $ useradd -g <groupname> <username>
All: $ passwd <username></code></pre><h4><br />
[2] Chanage permission of hadoop temp directory</h4><p>Change the permission of a directory in HDFS where hadoop stores its temporary data.<br />
Find that directory by looking entry <code>hadoop.tmp.dir</code> into <code>core-site.xml</code><br />
Then from the superuser account do the following step.</p><pre><code>$ hadoop fs –chmod -R 1777 /path/to/hadoop/tmp/directory</code></pre><h4><br />
[3] Give write access to hadoop temp directory in user machine</h4><p>The next step is to give write permission to our user group on <code>hadoop.tmp.dir</code>. Open <code>core-site.xml</code> to get the path for <code>hadoop.tmp.dir</code>. This should be done only in the machine(node) where the new user is added.</p><pre><code>$ chmod 777 /path/to/hadoop/tmp/dirctory</code></pre><h4><br />
[4] Create user home directory in HDFS</h4><p>The next step is to create a directory structure in HDFS for the new user.<br />
For that from the superuser, create a directory structure.</p><pre><code>$ hadoop fs –mkdir /user/username/</code></pre><h4><br />
[5] Change the ownership of user home directory in HDFS</h4><p>The ownership of the newly created directory structure is with superuser.With this new user<br />
will not be able to run mapreduce programs. So change the ownership of newly created directory in HDFS to the new user.</p><pre><code>$ hadoop fs –chown –R username:groupname <directory to access in HDFS>
Eg: hadoop fs –chown –R username:groupname /user/username/</code></pre><h4><br />
[6] Now run jobs with new user</h4><p>Login with new user and run jobs</p><pre><code>$ su - newuser
$ bin/hadoop jar pathToJar.jar mainClass input/path output/path</code></pre></div>Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com92tag:blogger.com,1999:blog-6353643695818271145.post-25722594888969002222014-04-07T20:26:00.000-07:002014-04-07T20:26:29.129-07:00Configuring OpenVPN Access Server and Connect Client in Ubuntu<div dir="ltr" style="text-align: left;color:black;" trbidi="on"><h3>OpenVPN Access Server consists of three major components:</h3><ol> <li>OpenVPN Server</li>
<li>Admin Web Interface/Admin UI</li>
<li>Connect Client</li>
</ol><br />
<h3>OpenVPN Server:</h3>The VPN server is the underlying component in OpenVPN Access Server that does all of the background work; routing, tunneling, encryption, user management, authentication etc. OpenVPN Access Server comes with a Web GUI that helps to manage the underlying components of the VPN server.<br />
<br />
<h3>Admin Web Interface:</h3><p>The Admin Web Interface makes for an easier management interface in OpenVPN Access Server. In the Admin Web Interface an administrator can manage options such as layer 2 or layer 3 routing, user permissions, server network settings, authentication and web server certificates. By default an administrator can access the Admin Web Interface by visiting this address in a web browser: https://openvpnasserverip/admin</p><br />
<h3>Connect Client:</h3><p>The Connect Client Interface is a component of OpenVPN Access Server that allows users to connect to the VPN directly through their web browser. The Connect Client also gives the user options to download their configuration files which can be userd on other OpenVPN clients.</p><br />
<h3>Installing the OpenVPN-AS Package:</h3><p>Download required version of OpenVPN AS from https://openvpn.net/index.php/access-server/download-openvpn-as-sw.html<br />
<code><br />
$ dpkg -i openvpnasdebpack.deb or [rpm -i openvpnasrpmpack.rpm]<br />
</code><br />
<br />
<h3>Configuring the Admin Web Interface:</h3><code><br />
$ passwd openvpn or (sometime $ /usr/local/openvpn_as/bin/ovpn-init)<br />
</code><br />
<p>Admin UI: https://YourIpAddress:943/admin<br />
Client UI: https://YourIPAddress:943/<br />
Browse https://YourIPAddress/admin and then configure other stuffs.</p><br />
Browse https://YourIPAddress<br />
<p>Download the 'OpenVPN Connect' software by clicking the link. After it has finished downloading, run it and enter your login credentials. And voilà! You are now connected to your OpenVPN Access Server.</p><p><strong>For that:</strong><br />
<h3>Installing an OpenVPN client:</h3><code><br />
$ sudo apt-get install openvpn [or sudo yum install openvpn]<br />
$ openvpn --version<br />
</code><br />
<br />
<strong>Start Client</strong><br />
<p>Login to the Access Server's Client Web Server and download the desired client config file (typically called "client.ovpn"<br />
<code><br />
$ openvpn --config client.ovpn<br />
</code></p></div>Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com9tag:blogger.com,1999:blog-6353643695818271145.post-74254881962945629072014-03-31T23:07:00.000-07:002017-03-06T00:19:02.277-08:00Configure Your Own Maven Repository using FTP/SCP/SSH in Ubuntu Server<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="color: black; text-align: left; text-size: 30px;" trbidi="on">
<h5>
Running your own Maven repository is dirt simple. All you need is a web server where you can upload files<br />
Maven supports FTP, SCP, WebDAV, and more exotic protocols like Subversion to publish artifacts to remote maven repository.<br />
</h5>
<h3>
[1] Install and Configure web server</h3>
<h5>
Install a web server (Apache HTTP Server or Nginx )<br />
</h5>
<code><br />
$ sudo apt-get install apache2 <br />
$ mkdir /var/www/maven-repo</code></div>
<br />
<h3>
[2] Install FTP server</h3>
<h5>
Install FTP server (VSFTPD)</h5>
<code><br />
$ sudo apt-ge install vsftpd<br />
$ sudo vim /etc/vsftpd.conf (edit on your behalf)<br />
$ sudo service vsftpd restart <br />
</code><br />
<h3>
[3] Configure Your Project Deployment</h3>
<h5>
Open up your awesome software project and edit the <i>pom.xml</i> file. You’re going to add two new sections, ending up with a file that looks like this:</h5>
<project ...=""><br />
...<br />
<groupid>test.maven</groupid><br />
<artifactid>awesome</artifactid><br />
<name>The Awesome Library</name><br />
<version>1.0-SNAPSHOT</version><br />
...<br />
<br />
<build><br />
...<br />
<extensions><br />
<extension><br />
<groupid>org.apache.maven.wagon</groupid><br />
<artifactid>wagon-ftp</artifactid><br />
<version>1.0-alpha-6</version><br />
</extension><br />
</extensions><br />
...<br />
</build><br />
<br />
...<br />
<distributionmanagement><br />
<repository><br />
<id>maven-repo</id><br />
<url>ftp://localhost/var/www/maven-repo</url><br />
</repository><br />
</distributionmanagement><br />
</project><br />
<h3>
[4] Configure Your Server Credentials</h3>
<h5>
You can find <i>settings.xml</i> in your personal Maven cache directory. On Unix-like systems, it should be at <i>~/.m2/settings.xml</i>. You may have to create the file if it doesn’t already exist. Here’s what it should contain:</h5>
<settings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemalocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"><br />
<servers><br />
<server><br />
<id>maven-repo</id><br />
<username>hacker</username><br />
<password>cracker</password><br />
</server><br />
</servers><br />
</settings><br />
<h5>
The <id> is the same as in our <i>pom.xml</i>. The user name and password are the credentials for your FTP server.</id></h5>
<h3>
[5] Deploy</h3>
<code><br />
$ mvn deploy</code><br />
<h3>
[6] Make your library available to user</h3>
<h5>
Now, anyone who wants to use your awesome library can just add a dependency to their pom.xml, like this:</h5>
...<br />
<dependencies><br />
...<br />
<dependency><br />
<groupid>test.maven</groupid><br />
<artifactid>awesome</artifactid><br />
<version>1.0-SNAPSHOT</version><br />
</dependency><br />
...<br />
<br />
</dependencies><br />
...<br />
<br />
<repositories><br />
...<br />
<repository><br />
<id>maven-repo</id><br />
<name>My Private Repository</name><br />
<url>http://localhost/maven-repo</url><br />
</repository><br />
...<br />
<br />
</repositories><br />
..<br />
<br />
<pre>Now have fun :)</pre>
</div>
Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com29tag:blogger.com,1999:blog-6353643695818271145.post-9831634248260204882014-03-24T01:41:00.000-07:002014-03-24T01:54:22.411-07:008 Ways to Backup Linux System<div dir="ltr" style="text-align: left;color: black" trbidi="on"><div dir="ltr" style="text-align: left;" trbidi="on"><div style="text-align: left;"><br />
</div><div style="text-align: left;"><h4><i><b><span style="font-weight: normal;">Data loss is an error condition in information systems in which information is destroyed by failures or neglect in storage, transmission, or processing.</span></b></i></h4></div><div style="text-align: left;"><h4><i><b><span style="font-weight: normal;">Information systems implement backup and disaster recovery equipment and processes to prevent data loss or restore lost data.More at <a href="http://en.wikipedia.org/wiki/Data_loss" target="_blank">Data Loss</a></span></b></i></h4></div><br />
<h3>[1] Using rsync:rsync stands for remote sync </h3><code> Syntax:<br />
$ rsync options source destination<br />
$ rsync -zvr /var/opt/installation/inventory/ /root/temp<br />
$ rsync -zvr /var/opt/installation/inventory/ hdhamee@remotehost.com:/root/temp<br />
where<br />
-z is to enable compression<br />
-v verbose<br />
-r indicates recursive</code><br />
<code><br />
</code><br />
<b>more at <a href="http://www.thegeekstuff.com/2010/09/rsync-command-examples/" target="_blank">rsync</a></b><br />
<br />
<h3>[2] Using rsnapshot<br />
</h3>Download the rsnapshot utility from <a href="http://www.rsnapshot.org/">rsnapshot.org</a> as shown below.<br />
<code><br />
$ cd /usr/src<br />
$ suodo wget http://www.rsnapshot.org/downloads/rsnapshot-1.3.1.tar.gz<br />
$ sudo tar xvfz rsnapshot-1.3.1.tar.gz<br />
$ cd rsnapshot-1.3.1<br />
$ ./configure --sysconfdir=/etc<br />
$ sudo make install</code><br />
<code><br />
</code><br />
<u><code>Configure:</code></u><br />
<code><code>$ sudo cp /etc/rsnapshot.conf.default /etc/rsnapshot.conf<br />
$ sudo vim /etc/rsnapshot.conf<br />
cmd_cp /bin/cp<br />
cmd_ssh /usr/bin/ssh<br />
cmd_du /usr/bin/du<br />
cmd_rsnapshot_diff /usr/local/bin/rsnapshot-diff<br />
logfile /var/log/rsnapshot<br />
backup /etc/ localhost/<br />
</code><br />
<u>Now test configuration:</u><br />
<code><br />
$ rsnapshot configtest<br />
Syntax OK<br />
</code><br />
<u>Verify the hourly backup configuration.</u><code><br />
$ sudo rsnapshot -t hourly<br />
</code><br />
<u>Verify the daily rsnapshot cwrsync backup process is configured properly.</u><code><br />
$ sudo rsnapshot -t daily<br />
</code><br />
<u>Add Crontab Entry for rsnapshot</u><code><br />
$ crontab -e<br />
0 */4 * * * /usr/local/bin/rsnapshot hourly<br />
30 23 * * * /usr/local/bin/rsnapshot daily<br />
</code><br />
</code></div><div style="text-align: left;"><u>Verify backup <code></code></u></div><code></code><br />
<code> $ sudo ls -al /.snapshots/<br />
ls: /.snapshots/: No such file or directory <br />
</code><br />
<br />
<br />
<h3>[3] Using dd Command [backup entire harddisk]</h3>To backup an entire copy of a hard disk to another hard disk connected to the same system<br />
<code><br />
$ dd if=/dev/sda of=/dev/sdb</code><br />
<code>Where:</code><br />
<code>if=input file</code><br />
<code>of=outputfile</code><br />
<h3>[4] Using dd Command [create an image of harddisk]</h3><code>$ sudo dd if=/dev/hda of=~/hdadisk.img<br />
</code><br />
<u><code>Restore using:</code></u><br />
<code>$ sudo dd if=hdadisk.img of=/dev/hdb</code><br />
<code><br />
Where:</code><br />
<code>if=input file</code><br />
<code>of=outputfile </code><br />
<h3>[5] Using dd Command [create a floppy image]</h3><code> $ sudo dd if=/dev/fd0 of=myfloppy.img</code><br />
<code><code>Where:</code></code><br />
<code>if=input file</code><br />
<code>of=outputfile </code><br />
<h3>[6] Using dd Command [backup a partition]</h3><code> $ sudo dd if=/dev/hda1 of=~/partition1.img</code><br />
<code><code>Where:</code></code><br />
<code>if=input file</code><br />
<code>of=outputfile </code><br />
<h3>[7] Using dd Command[CD ROM backup]</h3><code>$ sudo dd if=/dev/cdrom of=tgsservice.iso bs=2048 </code><br />
<code><code>Where:</code></code><br />
<code>if=input file</code><br />
<code>of=outputfile </code><br />
<h3>[8] Using Clonezilla </h3><h3><a href="http://clonezilla.org/" target="_blank">Clonezilla site</a></h3><br />
<code><br />
</code></div>Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com3tag:blogger.com,1999:blog-6353643695818271145.post-48216285970672778002014-03-13T00:06:00.002-07:002017-04-24T02:49:32.428-07:00Scala HelloWorld Program using SBT [Simple Build Tool]<div dir="ltr" style="text-align: left;" trbidi="on">
<h4 style="text-align: left;">
<span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;">SBT is a build tool for Scala, Java, and more. It requires Java 1.6 or later.</span></span></h4>
<span style="color: #660000;"><span style="font-family: "georgia" , "times new roman" , serif;"><br />
</span></span> <span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;"><u><b>Installation of SBT</b></u></span></span><br />
<span style="color: #660000;"><span style="font-family: "georgia" , "times new roman" , serif;"><code><br />
<span style="font-size: small;">$ mkdir $HOME/bin</span></code></span></span><br />
<span style="font-size: small;"><br /></span>
<span style="color: #660000;"><span style="font-family: "georgia" , "times new roman" , serif;"><code><span style="font-size: small;">$ wget http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.1/sbt-launch.jar -O $HOME/bin/sbt-launch.jar<br />
<br />
$ cat << EOM >> $HOME/bin/sbt<br />
SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"<br />
java $SBT_OPTS -jar $HOME/bin/sbt-launch.jar "$@"<br />
EOM<br />
<br />
$ chmod u+x $HOME//bin/sbt<br />
<br />
$ echo export PATH=$PATH:$HOME/bin >>$HOME/.bashrc</span><br />
</code></span></span><br />
<span style="color: #660000;"><span style="font-family: "georgia" , "times new roman" , serif;"><br />
</span></span> <span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;"><u><b>Create Scala Hello World Program</b></u></span></span><br />
<span style="color: #660000;"><span style="font-family: "georgia" , "times new roman" , serif;"><code><br />
$ mkdir scala-demo<br />
<br />
$ cd scala-demo<br />
<br />
$ echo 'object Hi { def main(args: Array[String]) = println("Hello World! Scala") }' > HelloWorld.scala<br />
$ sbt run<br />
<br />
output:>> <i>Hello World! Scala</i><br />
<br />
</code></span></span><br />
<span style="color: #660000;"><span style="font-family: "georgia" , "times new roman" , serif;"><br />
</span></span> <span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;">Moreover sbt works purely by convention. sbt will find the following automatically:</span></span><br />
<ul>
<li><span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;">Sources in the <b>base</b> directory</span></span></li>
<span style="font-size: large;">
</span>
<li><span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;">Sources in <b>src/main/scala</b> or <b>src/main/java</b></span></span></li>
<span style="font-size: large;">
</span>
<li><span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;">Tests in <b>src/test/scala</b> or <b>src/test/java</b></span></span></li>
<span style="font-size: large;">
</span>
<li><span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;">Data files in <b>src/main/resources</b> or <b>src/test/resources</b></span></span></li>
<span style="font-size: large;">
</span>
<li><span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;">jars in <b>lib</b></span></span></li>
</ul>
<span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;"><u><b>Common SBT commands</b></u></span></span><br />
<span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;">Here are some of the most common sbt commands. For a more complete list, see Command Line Reference.</span></span><br />
<ul>
<li><span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;"><b>clean</b> Deletes all generated files (in the target directory).</span></span></li>
<span style="font-size: large;">
</span>
<li><span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;"><b>compile</b> Compiles the main sources (in src/main/scala and src/main/java directories).</span></span></li>
<span style="font-size: large;">
</span>
<li><span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;"><b>test</b> Compiles and runs all tests.</span></span></li>
<span style="font-size: large;">
</span>
<li><span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;"><b>console</b> Starts the Scala interpreter with a classpath including the compiled sources and all dependencies. To return to sbt, type :quit, Ctrl+D (Unix), or Ctrl+Z (Windows).</span></span></li>
<span style="font-size: large;">
</span>
<li><span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;"><b>run</b> <argument>* Runs the main class for the project in the same virtual machine as sbt.</argument></span></span></li>
<span style="font-size: large;">
</span>
<li><span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;"><b>package</b> Creates a jar file containing the files in src/main/resources and the classes compiled from src/main/scala and src/main/java.</span></span></li>
<span style="font-size: large;">
</span>
<li><span style="color: #660000; font-size: large;"><span style="font-family: "georgia" , "times new roman" , serif;"><b>help</b> <command></command> Displays detailed help for the specified command. If no command is </span></span></li>
</ul>
</div>
Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com0tag:blogger.com,1999:blog-6353643695818271145.post-58934499941100518482014-02-01T14:17:00.001-08:002014-02-01T14:46:22.028-08:00Recompiling/Rebuilding Oracle VM VirtualBox Kernel Module<div dir="ltr" style="text-align: left; font-size:14px;" trbidi="on">After installing Oracle VM VirtualBox in Ubuntu or CentOS you may face very first problem while<br />
starting VirtualBox or even if started it may not function properly because It requires recompilation<br />
of Kernel module. <br />
<br />
==================================================================================<br />
<b>To recompile Kernel Module in Ubuntu or CentOS just got to terminal and type</b><br />
==================================================================================<br />
<br />
<i><b>[hikmat@dhcppc4 hdp2-ansible]$ sudo /etc/init.d/vboxdrv setup -v</i></b><br />
<br />
Stopping VirtualBox kernel modules [ OK ]<br />
Uninstalling old VirtualBox DKMS kernel modules [ OK ]<br />
Trying to register the VirtualBox kernel modules using DKMS<br />
Error! Your kernel headers for kernel 2.6.32-358.el6.x86_64 cannot be found at<br />
/lib/modules/2.6.32-358.el6.x86_64/build or /lib/modules/2.6.32-358.el6.x86_64/source.<br />
[FAILED]<br />
(Failed, trying without DKMS)<br />
Recompiling VirtualBox kernel modules [FAILED]<br />
(Look at /var/log/vbox-install.log to find out what went wrong)<br />
<br />
==================================================================================<br />
<b>To resolve this issue edit following file and add highlighted line</b>s<br />
==================================================================================<br />
<br />
<i><b>[hikmat@dhcppc4 hdp2-ansible]$ sudo vim /etc/init.d/vboxdrv</b></i><br />
#<br />
# chkconfig: 35 20 80<br />
# description: VirtualBox Linux kernel module<br />
#<br />
### BEGIN INIT INFO<br />
# Provides: vboxdrv<br />
# Required-Start: $syslog<br />
# Required-Stop:<br />
# Default-Start: 2 3 4 5<br />
# Default-Stop: 0 1 6<br />
# Short-Description: VirtualBox Linux kernel module<br />
### END INIT INFO<br />
<br />
PATH=/sbin:/bin:/usr/sbin:/usr/bin:$PATH<br />
DEVICE=/dev/vboxdrv<br />
LOG="/var/log/vbox-install.log"<br />
NOLSB=yes<br />
DEBIAN=<br />
VERSION=4.2.16<br />
MODPROBE=/sbin/modprobe<br />
<b>KERN_DIR=/usr/src/kernels/2.6.32-358.23.2.el6.x86_64</b><br />
<br />
if $MODPROBE -c | grep -q '^allow_unsupported_modules *0'; then<br />
MODPROBE="$MODPROBE --allow-unsupported-modules"<br />
fi<br />
<br />
[ -f /lib/lsb/init-functions ] || NOLSB=yes<br />
[ -f /etc/vbox/vbox.cfg ] && . /etc/vbox/vbox.cfg<br />
export BUILD_TYPE<br />
export USERNAME<br />
export USER=$USERNAME<br />
<b>export KERN_DIR</b><br />
"/etc/init.d/vboxdrv" 429L, 12698C <br />
<br />
<br />
===============================================================================<br />
<b>Compile Kernel Module again</b><br />
===============================================================================<br />
<br />
[hikmat@dhcppc4 hdp2-ansible]$ sudo /etc/init.d/vboxdrv setup -v<br />
<br />
Stopping VirtualBox kernel modules [ OK ]<br />
Uninstalling old VirtualBox DKMS kernel modules [ OK ]<br />
Trying to register the VirtualBox kernel modules using DKMS<br />
Error! Your kernel headers for kernel 2.6.32-358.el6.x86_64 cannot be found at<br />
/lib/modules/2.6.32-358.el6.x86_64/build or /lib/modules/2.6.32-358.el6.x86_64/source.<br />
[FAILED]<br />
(Failed, trying without DKMS)<br />
Recompiling VirtualBox kernel modules [ OK ]<br />
Starting VirtualBox kernel modules [ OK ]<br />
[hikmat@dhcppc4 hdp2-ansible]$ virtualbox <br />
<br />
===============================================================================<br />
<b>Then start virtual machine again and enjoy :) :)</b><br />
==============================================================================<br />
$ sudo virtualbox<br />
<br />
it will start without errors<br />
<br />
<br />
</div>Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com25tag:blogger.com,1999:blog-6353643695818271145.post-1551561281901984862013-12-30T02:53:00.002-08:002013-12-30T02:53:27.671-08:00How to Add Facebook Login Button onto WebPage<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="font-size: 14px;">
<br /></div>
<div style="font-size: 14px;">
<b> Facebook offers a lots of social plugins. Here we are going to<br />
add a Login Button (Login Button Plugin) onto webpage:</b></div>
<div style="font-size: 14px;">
<br /></div>
<div style="font-size: 14px;">
<br /></div>
<div style="font-size: 14px;">
<b>[1]</b> Go to https://developers.facebook.com/apps/ and click on the "Create New App"</div>
<div style="font-size: 14px;">
button or "Register as Developer", in case of new user,follow on-screen steps and</div>
<div style="font-size: 14px;">
remember the App Id code for later use.</div>
<div style="font-size: 14px;">
<br /></div>
<div style="font-size: 14px;">
<br /></div>
<div style="font-size: 14px;">
<b>[2]</b> Go to http://developers.facebook.com/docs/reference/plugins/login/ and click</div>
<div style="font-size: 14px;">
the "Get Code" button.And you will get code like </div>
<div style="font-size: 14px;">
<br /></div>
<pre style="background-color: #f0f0f0; border: 1px dashed rgb(204, 204, 204); height: auto; overflow: auto; padding: 0px; text-align: left; width: 99%;"><span style="background-color: transparent; font-size: 12px; line-height: 20px;">
</span></pre>
<pre style="background-color: #f0f0f0; border: 1px dashed rgb(204, 204, 204); height: auto; overflow: auto; padding: 0px; text-align: left; width: 99%;">
</pre>
<pre style="background-color: #f0f0f0; border: 1px dashed rgb(204, 204, 204); height: auto; overflow: auto; padding: 0px; text-align: left; width: 99%;"><span style="font-family: arial; font-size: 12px; line-height: 20px;">
</span></pre>
<pre style="background-color: #f0f0f0; border: 1px dashed rgb(204, 204, 204); height: auto; overflow: auto; padding: 0px; text-align: left; width: 99%;"><span style="font-family: arial; font-size: 12px; line-height: 20px;">
</span></pre>
</div>
<div style="font-size: 14px;">
<b>[3]</b> Paste above HTML5 code just below the body tag and replace <b>APP_ID</b> with the </div>
<div style="font-size: 14px;">
app id you created for your Facebook app.</div>
<div style="font-size: 14px;">
<br /></div>
<div style="font-size: 14px;">
<b>[4]</b> Now paste the code from the second box in whatever place you want the login button</div>
<div style="font-size: 14px;">
to reside. The code should look something like:</div>
<div style="font-size: 14px;">
<br /></div>
<div style="font-size: 14px;">
<br /></div>
<pre style="background-color: #f0f0f0; background-position: initial initial; background-repeat: initial initial; border: 1px dashed rgb(204, 204, 204); color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">
<div class="fb-login-button" data-max-rows="1" data-show-faces="false" data-width="200">
</div>
</code></pre>
<div style="font-size: 14px;">
<b>[5]</b> Open you page in browser and Enjoy!!! with Facebook Login Button.</div>
</div>
Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com3tag:blogger.com,1999:blog-6353643695818271145.post-46073669436560011322013-12-22T08:14:00.000-08:002013-12-22T08:14:28.081-08:00Mounting NTFS Drive in CentOS 6.4 (Simple and Easy)<div dir="ltr" style="text-align: left;" trbidi="on"><b>Some time mounting the NTFS drive in CentOS is tedious process.<br />
But now the stone is too soft and you can break it easily.<br />
Just follow the following steps.</b><br />
<br />
<i>$ sudo yum install fuse fuse-ntfs-3g</i><br />
<br />
<b>Than just click on NTFS drive you want mount<br />
<br />
Enjoy!!!</b><br />
<br />
</div>Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com2tag:blogger.com,1999:blog-6353643695818271145.post-86952371794688375792013-12-16T08:16:00.000-08:002013-12-16T08:31:05.875-08:00Ganglia Architecture and Module Development<b><div dir="ltr" style="text-align: left;" trbidi="on"><div style="margin-bottom: 0in; margin-right: 0.15in;font-size:large;"><br />
<br />
</div><div style="margin-bottom: 0in; margin-right: 0.15in;"><span style="color: navy;"><span style="background: #008000;"><br />
<span style="font-family: Currior new;"><span style="font-size: medium;"><b>Ganglia<br />
Architecture and Module Development </b></span></span></span></span></div><br />
<div style="margin-bottom: 0in;"><span style="font-family: Currior new;">Ganglia<br />
monitoring suite consists of three main parts: gmond, gmetad and<br />
ganglia-web. </span></div><div style="margin-bottom: 0in;"><br />
<br />
</div><table cellpadding="2" cellspacing="0" style="width: 743px;"><colgroup><col width="70"></col> <col width="665"></col> </colgroup><tbody>
<tr> <td style="border: none; padding: 0in;" width="70"><br />
<b><span style="font-family: Currior new;"><span style="font-size: x-small;">gmond</span></span></b><span style="font-family: Currior new;"><span style="font-size: x-small;"><br />
</span></span></td> <td style="border: none; padding: 0in;" width="665"><br />
<span style="font-family: Currior new;"><span style="font-size: x-small;">The ganglia monitoring<br />
daemon (gmond) is a lightweight service that is installed on every<br />
machine you'd like to monitor. This daemon uses a simple<br />
listen/announce protocol via XDR to collect monitoring state and<br />
then shares this information via XML over TCP. Gmond is portable<br />
and collects dozens of system metrics: CPU, memory, disk, network<br />
and process data. </span></span></td> </tr>
<tr> <td style="border: none; padding: 0in;" width="70"><br />
<b><span style="font-family: Currior new;"><span style="font-size: x-small;">gmetad</span></span></b><span style="font-family: Currior new;"><span style="font-size: x-small;"><br />
</span></span></td> <td style="border: none; padding: 0in;" width="665"><br />
<span style="font-family: Currior new;"><span style="font-size: x-small;">The ganglia meta daemon<br />
(gmetad) is a service that collects data from other gmetad and<br />
gmond sources and stores their state to disk in indexed<br />
round-robin databases. Gmetad provides a simple query mechanism<br />
for collecting specific information about groups of machines.<br />
Gmetad supports hierarchical delegation for creating manageable<br />
monitoring domains. </span></span></td> </tr>
<tr> <td style="border: none; padding: 0in;" width="70"><br />
<b><span style="font-family: Currior new;"><span style="font-size: x-small;">gmetric</span></span></b><span style="font-family: Currior new;"><span style="font-size: x-small;"><br />
</span></span></td> <td style="border: none; padding: 0in;" width="665"><br />
<span style="font-family: Currior new;"><span style="font-size: x-small;">The ganglia metric tool<br />
is a commandline application that you can use to inject custom<br />
made metrics about hosts that are being monitored by ganglia. It<br />
has the ability to spoof messages as coming from a different host<br />
in case you want to capture and report metrics from a device where<br />
you don't have gmond running (like a network or other embedded<br />
device). </span></span></td> </tr>
<tr> <td style="border: none; padding: 0in;" width="70"><br />
<b><span style="font-family: Currior new;"><span style="font-size: x-small;">gstat</span></span></b><span style="font-family: Currior new;"><span style="font-size: x-small;"><br />
</span></span></td> <td style="border: none; padding: 0in;" width="665"><br />
<span style="font-family: Currior new;"><span style="font-size: x-small;">The ganglia stat tool is<br />
a commandline application that you can use to query a gmond for<br />
metric information directly. </span></span></td> </tr>
<tr> <td style="border: none; padding: 0in;" width="70"><br />
<b><span style="font-family: Currior new;"><span style="font-size: x-small;">gweb</span></span></b><span style="font-family: Currior new;"><span style="font-size: x-small;"><br />
</span></span></td> <td style="border: none; padding: 0in;" width="665"><br />
<span style="font-family: Currior new;"><span style="font-size: x-small;">The ganglia web frontend<br />
expresses the data stored by gmetad in a graphical web interface<br />
using PHP. </span></span></td> </tr>
</tbody></table><div style="margin-bottom: 0in;"><img align="LEFT" border="0" src="http://1.bp.blogspot.com/-6p36g3F8Wvg/Uq8pvAx0CzI/AAAAAAAAAGc/IjmvNus-SXI/s320/ganglia_html_m6e02aebe.png" height="483" name="graphics1" width="696" /><br />
<span style="font-family: Currior new;">Fig. Ganglia<br />
Architecture in Cluster Mode</span></div><div style="margin-bottom: 0in;"></div><div style="margin-bottom: 0in;"><span style="color: navy;"><span style="background: #008000;"><br />
<span style="font-family: Currior new;"><span style="font-size: medium;"><b>Ganglia-web Extension <br />
</b></span></span></span></span></div><br />
<div style="margin-bottom: 0in;"><span style="font-family: Liberation Serif, serif;"><span style="font-size: x-small;"><span style="font-family: Currior new;">All<br />
of the "</span><span style="font-family: Currior new;"><b>graph handlers</b></span><span style="font-family: Currior new;">"<br />
are stored in the ../</span><span style="font-family: Currior new;"><b>ganglia</b></span><tt class="western"><span style="font-family: Currior new;"><b>/graph.d</b></span></tt><span style="font-family: Currior new;"><b><br />
</b></span><span style="font-family: Currior new;">directory, where "ganglia"<br />
is the top-level directory used by the webserver to publish the<br />
various Ganglia webpages. Ganglia ships with </span><span style="font-family: Currior new;"><span style="font-weight: normal;">one<br />
general handler for </span></span><span style="font-family: Currior new;"><b>metrics</b></span><span style="font-family: Currior new;"><span style="font-weight: normal;">,<br />
</span></span><span style="font-family: Currior new;"><b>five basic reports</b></span><span style="font-family: Currior new;"><span style="font-weight: normal;">,<br />
and one heavily commented </span></span><span style="font-family: Currior new;"><b>sample<br />
report</b></span><span style="font-family: Currior new;"><span style="font-weight: normal;">.</span></span></span></span></div><ul><ul><li><div style="font-weight: normal; margin-bottom: 0in;"><span style="font-family: Currior new;"><span style="font-size: x-small;">The<br />
default layout of ganglia-web and default view of graphs can be<br />
modified according our need. For that we just need to modify some<br />
PHP code.</span></span></div></li>
<li><div style="font-weight: normal; margin-bottom: 0in;"><span style="font-family: Liberation Serif, serif;"><span style="font-size: x-small;"><span style="color: #3c3c41;"><span style="text-decoration: none;"><span style="font-family: Currior new;"><span style="font-style: normal;"><span style="text-decoration: none;">Uses<br />
presentation templates so that the web site “look </span></span></span></span></span><span style="color: #3c3c41;"><span style="text-decoration: none;"><span style="font-family: Currior new;"><span style="font-style: normal;"><span style="text-decoration: none;"><br />
and feel” can be easily customized.</span></span></span></span></span></span></span></div></li>
</ul></ul><ul><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; text-decoration: none;"><span style="color: #3c3c41;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;">modify ../ganglia/templates/defaults/*.tpl</span></span></span></span></div></ul><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; text-decoration: none;"><br />
<br />
</div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; text-decoration: none;"><br />
<br />
</div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; text-decoration: none;"><br />
<br />
</div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; text-decoration: none;"><br />
<br />
</div><ul><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- Adding Custom Reports</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">It is possible to add new custom reports, without making any changes to the “Official” Ganglai PHP code. Adding new report is a four step process.</span></span></span></span></div></ul><ol><div style="margin-bottom: 0in;"><span style="font-family: Currior new;"><span style="font-size: x-small;">1. All reports should be contained in a single PHP file with the name </span></span><tt class="western"><span style="font-family: Currior new;"><span style="font-size: x-small;">"reportName_report.php"</span></span></tt><span style="font-family: Currior new;"><span style="font-size: x-small;"> in the </span></span> </div><div style="margin-bottom: 0in;"><tt class="western"> </tt><tt class="western"><span style="font-family: Currior new;"><span style="font-size: x-small;">graph.d/</span></span></tt><span style="font-family: Currior new;"><span style="font-size: x-small;"> directory. Obviously, replace "reportName" with something suitable.</span></span></div><div style="margin-bottom: 0in;"><span style="font-family: Currior new;"><span style="font-size: x-small;">2. The *_report function should return an associative array that contains a number of specifically named </span></span> </div><div style="margin-bottom: 0in;"><span style="font-family: Currior new;"><span style="font-size: x-small;">variables example:<b>$series,$title,$vertical_label </b>etc.</span></span></div><span style="font-family: Currior new;"><span style="font-size: x-small;">3. Within this file, you <i>must</i> have a PHP function called "reportName_report" that takes a single array argument that is passed by reference (<i>not</i> by value!). The function definition for this will look something like this: </span></span> </ol><pre class="western" style="margin-left: 1.08in;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>function temperature_report ( &$rrdtool_graph ) {</b></i></span></span>
<span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>$rrdtool_graph['vertical_label'] = 'Temperature (C)';</b></i></span></span>
<span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>$rrdtool_graph['title'] = 'CPU Temeratures';</b></i></span></span>
<span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>$rrdtool_graph['series'] = "... very long set of RRD tool RPN commands...";</b></i></span></span>
<span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>}</b></i></span></span></pre><ol start="0"><div style="margin-bottom: 0in;"><span style="font-family: Liberation Serif, serif;"><span style="font-size: x-small;"><span style="font-family: Currior new;">4. Edit the </span><span style="font-family: Currior new;"><i><b>$optional_graphs</b></i></span><span style="font-family: Currior new;"><b> </b></span><span style="font-family: Currior new;">variable in the </span><tt class="western"><span style="font-family: Currior new;">conf.php</span></tt><span style="font-family: Currior new;"> file to include the base name of your new report. </span></span></span> </div><div style="margin-bottom: 0in;"><span style="font-family: Currior new;"><span style="font-size: x-small;">Thus, if the function name is reportName_report, and the PHP file is "reportName_report.php", include </span></span> </div><div style="margin-bottom: 0in;"><span style="font-family: Currior new;"><span style="font-size: x-small;">only "reportName" in the $optional_graphs array. </span></span> </div></ol><pre class="western" style="margin-bottom: 0.2in; margin-left: 0.57in;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><b>$optional_graphs = array('packets', 'reportName');</b></span></span></pre><div style="margin-bottom: 0in;"><span style="text-decoration: none;"> </span><span style="font-family: Currior new;"><span style="font-size: x-small;"><u>Sample report to view cpu user</u><br />
<span style="font-size: xx-small;"><i><b><?php </b></i></span></span></span> </div><div style="margin-bottom: 0in; margin-left: 0.67in;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>function graph_demo_report ( &$rrdtool_graph ) { </b></i></span></span> </div><div style="margin-bottom: 0in; margin-left: 0.67in;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>global $cpu_user_color, </b></i></span></span> </div><div style="margin-bottom: 0in; margin-left: 0.67in;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>$rrd_dir; </b></i></span></span> </div><div style="margin-bottom: 0in; margin-left: 0.67in;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>$rrdtool_graph['title'] ='==Demo Report=='; </b></i></span></span> </div><div style="margin-bottom: 0in; margin-left: 0.67in;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>$rrdtool_graph['vertical-label'] = '==Demo Report=='; </b></i></span></span> </div><div style="margin-bottom: 0in; margin-left: 0.67in;"><br />
</div><div style="margin-bottom: 0in; margin-left: 0.67in;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>$series ="DEF:'cpu_user'='${rrd_dir}/cpu_user.rrd':'sum':AVERAGE " </b></i></span></span> </div><div style="margin-bottom: 0in; margin-left: 0.67in;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>. "AREA:'cpu_user'#$cpu_user_color:'==CPU User==' "; </b></i></span></span> </div><div style="margin-bottom: 0in; margin-left: 0.67in;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>$rrdtool_graph['series'] = $series; </b></i></span></span> </div><div style="margin-bottom: 0in; margin-left: 0.67in;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>return $rrdtool_graph; </b></i></span></span> </div><div style="margin-bottom: 0in; margin-left: 0.67in;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>} </b></i></span></span> </div><div style="margin-bottom: 0in;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>?></b></i></span></span></div><div style="margin-bottom: 0in;"><br />
</div><div style="margin-bottom: 0in; text-decoration: none;"><span style="color: navy;"><span style="background: #008000;"> <span style="font-family: Currior new;"><span style="font-size: medium;"><b>Gmond Matrix Gathering Agent Extension </b></span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.14in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;"><span style="color: #3c3c41;"><span style="text-decoration: none;"><span style="font-style: normal;"><span style="text-decoration: none;">- </span></span></span></span><span style="color: #3c3c41;"><span style="text-decoration: none;"><span style="font-style: normal;"><b><span style="text-decoration: none;">Gmetric</span></b></span></span></span><span style="color: #3c3c41;"><span style="text-decoration: none;"><span style="font-style: normal;"><span style="text-decoration: none;"> – Out-of-process utility capable of invoking command </span></span></span></span><span style="color: #3c3c41;"><span style="text-decoration: none;"><span style="font-style: normal;"><span style="text-decoration: none;"> line based metric gathering scripts</span></span></span></span></span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.14in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- <b>Loadable modules </b>capable of gathering multiple metrics or using advanced metric gathering APIs</span></span></span></span></div><div align="LEFT" style="font-style: normal; line-height: 88%; margin-bottom: 0in; text-decoration: none;"><span style="color: #333333;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><u><b><span style="text-decoration: none;">Gmond Pluggable Metric Modules Development</span></b></u></span></span></span></div><div align="LEFT" style="font-style: normal; line-height: 88%; margin-bottom: 0in; text-decoration: none;"><span style="color: #e00000;"><span style="text-decoration: none;"> <span style="color: black;"> <span style="font-family: Currior new;"><span style="font-size: x-small;"><b>- Using C</b></span></span></span></span></span></div><div align="LEFT" style="font-style: normal; line-height: 88%; margin-bottom: 0in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;"><b>- Using Python</b></span></span></span></span></div><div align="LEFT" style="font-style: normal; line-height: 88%; margin-bottom: 0in; text-decoration: none;"><br />
</div><div align="LEFT" style="font-style: normal; line-height: 88%; margin-bottom: 0in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="background: #008000;"> <span style="font-family: Currior new;"><span style="font-size: small;"><b>Using Python</b></span></span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.53in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- Extends the available metrics that can be gathered by Gmond</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.53in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- Provided as dynamically loadable modules</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.53in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- Configured through the gmond.conf</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.53in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- Scheduled through Gmond rather than an external scheduler</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.53in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- Module development is similar to an Apache module</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.53in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- Able to produce multiple metrics from a single module</span></span></span></span></div><div align="LEFT" style="font-style: normal; line-height: 97%; margin-bottom: 0.12in; text-decoration: none;"><span style="color: #3c3c41;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;"><b>Each module must contain three mandatory functions</b></span></span></span></span></div><div align="LEFT" style="font-style: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><b><span style="text-decoration: none;">metric_init()</span></b></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 100%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- Called once at module initialization time</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 100%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- Must return a metric description dictionary or list of dictionaries</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 100%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- Any other module initialization can also take place here</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;"><b>metric_handler() </b>–</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #3c3c41;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;">- may have multiple handlers</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #3c3c41;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;">- Metric gathering handler</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- Must return a single data value of the same type as specified in the metric_init() function</span></span></span></span></div><div align="LEFT" style="font-style: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><b><span style="text-decoration: none;">metric_cleanup()</span></b></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">- Called once at module termination time</span></span></span></span></div><ul><li><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">Does<br />
not return a value</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; text-decoration: none;"><span style="color: black;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;">Example<br />
Module:</span></span></span></span></div></li>
</ul><div align="LEFT" style="font-style: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 1.32in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: medium;"><b><span style="font-size: xx-small;"><i>Curve_Max = 15 </i></span></b></span></span></span></span></div><div align="LEFT" style="line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>v = int(1)</b></i></span></span></span></span></div><div align="LEFT" style="line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>inc = int(1)</b></i></span></span></span></span></div><div align="LEFT" style="line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>count = 0</b></i></span></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><br />
</div><div align="LEFT" style="line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>def metric_init(params):</b></i></span></span></span></span></div><div align="LEFT" style="line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>global Curve_Max</b></i></span></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><br />
</div><div align="LEFT" style="line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>if ‘CurveMax’ in params:</b></i></span></span></span></span></div><div align="LEFT" style="line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>Curve_Max = int(params[‘CurveMax’])</b></i></span></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><br />
</div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>d = {‘name’: ‘Curve_Metric’, </b></span></span></i></span></span></span> </div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>‘call_back’: curve_handler, </b></span></span></i></span></span></span> </div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>‘time_max': int(60),</b></span></span></i></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>‘value_type’: ‘uint’, </b></span></span></i></span></span></span> </div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>‘units’: ‘Seconds’,</b></span></span></i></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>‘slope’: ‘both’, </b></span></span></i></span></span></span> </div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>‘format’: ‘%u’,</b></span></span></i></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>‘description’: </b></span></span></i></span></span></span> </div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b> ‘Shows a uniform curve’}</b></span></span></i></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><br />
</div><div align="LEFT" style="line-height: 90%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.01in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>return d</b></i></span></span></span></span></div><div align="LEFT" style="line-height: 97%; margin-bottom: 0.12in; margin-left: 1.32in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>def curve_handler(name):</b></i></span></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 105%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.04in; text-decoration: none;"><span style="color: #3c3c41;"><span style="text-decoration: none;"><span style="color: black;"> </span><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i> <span style="color: black;"><b>global v,count,inc,Curve_Max</b></span></i></span></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 105%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.04in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>v += inc</b></span></span></i></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 105%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.04in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>count += 1</b></span></span></i></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 105%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.04in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>if count > Curve_Max:</b></span></span></i></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 105%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.04in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>count = 0</b></span></span></i></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 105%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.04in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>inc = -inc</b></span></span></i></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 105%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.04in; text-decoration: none;"><br />
</div><div align="LEFT" style="font-weight: normal; line-height: 105%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.04in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>return int(v)</b></span></span></i></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 105%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.04in; text-decoration: none;"><br />
</div><div align="LEFT" style="line-height: 105%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.04in; text-decoration: none;"><span style="color: black;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><b>def metric_cleanup():</b></i></span></span></span></span></div><div align="LEFT" style="font-weight: normal; line-height: 105%; margin-bottom: 0in; margin-left: 1.32in; margin-top: 0.04in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: xx-small;"><i><span style="text-decoration: none;"> <span style="color: black;"><b>pass</b></span></span></i></span></span></span></div><div align="LEFT" style="font-style: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #333333;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><u><b><span style="text-decoration: none;">Gmond Python Module Deployment</span></b></u></span></span></span></div><div align="LEFT" style="font-style: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #e00000;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;"><b>- </b><span style="color: #333333;"><span style="font-weight: normal;">Copy the .py file to python directory specified in gmond.conf file put in gmond.conf</span></span></span></span></span></span></div><div align="LEFT" style="line-height: 97%; margin-bottom: 0.12in; margin-left: 1.83in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><i><b><span style="text-decoration: none;">modules {</span></b></i></span></span></span></div><div align="LEFT" style="line-height: 97%; margin-bottom: 0.12in; margin-left: 1.83in; text-decoration: none;"><span style="color: #3c3c41;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;"><i><b>module {</b></i></span></span></span></span></div><div align="LEFT" style="line-height: 97%; margin-bottom: 0.12in; margin-left: 1.83in; text-decoration: none;"><span style="color: #3c3c41;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;"><i><b>name = "python_module"</b></i></span></span></span></span></div><div align="LEFT" style="line-height: 70%; margin-bottom: 0in; margin-left: 1.83in; margin-top: 0.04in; text-decoration: none;"><span style="color: #3c3c41;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;"><i><b>path = "/usr/lib/ganglia/modpython.so"</b></i></span></span></span></span></div><div align="LEFT" style="line-height: 70%; margin-bottom: 0in; margin-left: 1.83in; margin-top: 0.04in; text-decoration: none;"><span style="color: #3c3c41;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;"><i><b>params = "/usr/lib/ganglia/python_modules"</b></i></span></span></span></span></div><div align="LEFT" style="line-height: 70%; margin-bottom: 0in; margin-left: 1.83in; margin-top: 0.04in; text-decoration: none;"><span style="color: #3c3c41;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;"><i><b>}</b></i></span></span></span></span></div><div align="LEFT" style="line-height: 70%; margin-bottom: 0in; margin-left: 1.83in; margin-top: 0.04in; text-decoration: none;"><span style="color: #3c3c41;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><i><b><span style="text-decoration: none;">}</span></b></i></span></span></span></div><div align="LEFT" style="line-height: 97%; margin-bottom: 0.12in; margin-left: 1.83in; text-decoration: none;"><br />
<br />
</div><ol start="2"><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; text-decoration: none;"><span style="color: #333333;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;">- Define the metric to collection group for this module</span></span></span></span></div></ol><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; text-decoration: none;"><span style="color: #333333;"><span style="text-decoration: none;"> <span style="font-family: Currior new;"><span style="font-size: x-small;">- Restart gmond</span></span></span></span></div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><br />
<br />
</div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><br />
<br />
</div><div align="LEFT" style="font-style: normal; font-weight: normal; line-height: 97%; margin-bottom: 0.12in; margin-left: 0.93in; text-decoration: none;"><span style="color: #333333;"><span style="font-family: Currior new;"><span style="font-size: x-small;"><span style="text-decoration: none;"></span></span></span></span></div></div></b>Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com18tag:blogger.com,1999:blog-6353643695818271145.post-28431375348636722762013-08-14T05:44:00.002-07:002014-03-21T05:28:04.908-07:00Version Automation in Grails Application <div dir="ltr" style="text-align: left;" trbidi="on"><h4>To increment version of grails app in every build or compile</h4><ol><li>Add _Events.groovy file in PROJECT_HOME/scripts folder</li>
<li>Edit file and add following code</li>
</ol><code><br />
eventCompileStart = { kind -><br />
def buildNumber = metadata.'app.buildNumber'<br />
<br />
if (!buildNumber)<br />
buildNumber = 1<br />
else<br />
buildNumber = Integer.valueOf(buildNumber) + 1<br />
<br />
metadata.'app.buildNumber' = buildNumber.toString()<br />
<br />
metadata.persist()<br />
<br />
println "**** Compile Starting on Build #${buildNumber}"<br />
</code><br />
</div>Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com0tag:blogger.com,1999:blog-6353643695818271145.post-47300750866980536842013-07-12T07:16:00.001-07:002014-03-21T05:21:03.985-07:00Protobuff installation in Linux<div dir="ltr" style="text-align: left;" trbidi="on"><h3>Pre Installation Tasks</h3><h5>Remove older version of protobuff before installing newer version</h5><code><br />
$ sudo apt-get remove protobuf-compiler<br />
$ sudo ldconfig<br />
$ sudo apt-get install g++ (if not already installed)<br />
Confirm g++ installation:<br />
$ g++ --version<br />
</code><br />
<h3>Installation Tasks</h3><code><br />
$ cd ~/Downloads (or where ever you want store)<br />
$ wget http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz<br />
$ tar xzf protobuf-2.5.0.tar.gz<br />
$ cd protobuf-2.5.0<br />
$ ./configure<br />
$ make<br />
$ sudo make install<br />
</code><br />
<h3>Post Installation Tasks</h3><code><br />
$ sudo ldconfig<br />
Confirm installation:<br />
$ protoc --version<br />
</code></div>Rhitikhttp://www.blogger.com/profile/07340627089735445864noreply@blogger.com0