Francisco Lopez

From InterSciWiki
Jump to: navigation, search

HPC@UCI

Hi Doug

  • Just an FYI that I've created this account for use with the socscicompute.ss.uci.edu website. I've been my own account to test porting the backend processing from XSEDE to UCI's local HPC cluster for the development site. I've now created this account to handle link the two systems.
  • CoSSci -- your HPC Cluster account is ready.
  • I'll contact you again when I'm ready to migrate the production site.
  • Please let me know if you have any questions.
  • Sincerely,
  • -Francisco Lopez CoSSci

-- AWS -- UCI OIT's data center -- http://www.oit.uci.edu/oit-data-center/

Francisco Lopez, May 18 2017

Hi Doug,

Just an FYI that I've created this account for use with the socscicompute.ss.uci.edu website. I've been my own account to test porting the backend processing from XSEDE to UCI's local HPC cluster for the development site. I've now created this account to handle link the two systems.

I'll contact you again when I'm ready to migrate the production site.

Please let me know if you have any questions.

Sincerely,

-Francisco


On Wed, May 17, 2017 at 10:25 AM, <hpc-support@uci.edu> wrote:

   Greetings.
   This email was sent to notify you that your HPC Cluster account is ready.
   It is mandatory that ALL NEW HPC users read the following HPC web page before
   running jobs:
           http://hpc.oit.uci.edu/running-jobs


   Due to security issues, the cluster can only be accessed from a machine on the
   UCI network:  VPN or on-campus machines on the .uci.edu domain.
   You will need to use SSH or an equivalent secure login method such as x2go to
   access the cluster.
   The HPC Cluster name is: hpc.oit.uci.edu
   To login using ssh:
           ssh  drwhite@hpc.oit.uci.edu
   To enable X11 ( if you want GUI to display ):
           ssh -X -Y drwhite@hpc.oit.uci.edu
   Your password is your UCInetID password.  If you forgot your UCInetID password,
   you can reset at:  http://www.oit.uci.edu/ucinetid/ucinetid-password.html


   HPC Main web page can be found at:
            http://hpc.oit.uci.edu
   To see what queues ( private / public ) you have access to, enter:
           queue
   at the system prompt.   You can also use 'q' for short.


   If you do not have a secure method of connecting to HPC, we recommend using
   x2go.  You may download your x2go client from:
          http://www.x2go.org/doku.php/download:start
   If you are having issues accessing the HPC Cluster, please email the HPC
   support team at hpc-support@uci.edu
   Note: It may take up to 5 minutes for your account to be activated.
   Welcome to the UCI campus High Performance Cluster,
   The HPC Team
   _______________________________________________
   hpc-support mailing list
   hpc-support@uci.edu
   https://maillists.uci.edu/mailman/listinfo/hpc-support

Francisco Lopez, May 17 2017 10 pages in print

New User Warnings

	1) If you have never used a Cluster, or are not familiar with this cluster, YOU WILL WANT to read and follow the examples below to become familiar with how to run jobs on HPC. It is a common practice by new users to ignore this manual and simply try to run jobs without understanding what they are doing. Such carelessness can and WILL easily impact hundreds/thousands of critical jobs and users currently running on the cluster. If your actions compromise the health of the HPC cluster, your account will be LOCKED so please make sure you run through the examples below before you embark on running jobs.
	2) Do NOT use the login nodes for work. If everyone does this, the login nodes will crash keeping 700+ HPC users from being able to login to the cluster.
	3) Do NOT use your home-directory (/data/users/$USER) for any serious work. See http://hpc.oit.uci.edu/data-storage
	4) Never submit large number of jobs (greater than 5) without first running a small test case to make sure all works as expected. Start slow and then ramp up with jobs once you are familiar with how things work.
	5) We have a SKELETON crew running the UCI HPC cluster. Before you use the cluster and/or ask for help, PLEASE make sure you run the Serial Job example explained below before contacting HPC support.
  • How to use HPC
  • Using a High Performance Computing Cluster such as the UCI HPC Cluster requires at a minimum some basic understanding of the Linux Operating System.
  • It is outside the scope of this manual to explain Linux commands and/or how parallel programs such as MPI work. This manual simply explains how to run jobs on the HPC cluster.
  • When you login to hpc.oit.uci.edu, you are connected to what is called a login node. The HPC Cluster has several major components:

•* A Head Node •* Login Nodes ( hpc.oit.uci.edu ) •* Interactive Nodes •* Compute Nodes •* I/O Nodes •* Data Servers

  • The head node runs all of the cluster critical services. The traditional cluster "Head Node" on this cluster is hidden. It is hidden from users for two major reasons. To lesson external Internet attacks and to prevent accidental user’s jobs gone-wrong from taking entire cluster down with them :-)
  • The login node hpc.oit.uci.edu is the node you get when you first log into HPC. The login nodes are meant for simple tasks such as submitting jobs, checking on job status, editing (emacs, vi) and performing simple tasks.
    • The interactive nodes are used for when you need to compile, test your code, and run a few 1-2 interactive sessions.
    • The compute nodes are the workhorse of the cluster. For computational work both Serial or Parallel, in Batch mode or Interactive mode, you will be using the compute nodes.
    • The I/O nodes (ionodes) are used to transfer data to & from HPC. When you need to transfer a lot of data, you will be using an ionode.
    • The Data Servers are just that. For a complete list of all Data Servers available on HPC, please see:

•** http://hpc.oit.uci.edu/data-storage

    • Grid Engine Scheduler
    • The HPC cluster is using Son of Grid Engine (GE for short), to manage all of the resources (the nodes) on the cluster.
    • GE provides user commands like qsub, qdel, qstat and qrsh which are used to submit, delete, check the status of the jobs and to request interactive nodes on the cluster.
    • Learning how to use Grid Engine can be a major challenge for those who have never used it before, so we will explain only the basics here and only enough for you to get jobs running on HPC. If you like to **learn more about Grid Engine, there are lots of good examples on the web. A couple of links are:

•* How To Use Grid Engine •* Grid Engine Reference Sheet Here is a list of the most common GE commands you will be using:

10 pages more

May 17 2017 HPC Francisco Lopez, Harry J. Mangalam

http://hpc.oit.uci.edu

Hi Doug,

Just an FYI that I've created this account for use with the socscicompute.ss.uci.edu website. I've been my own account to test porting the backend processing from XSEDE to UCI's local HPC cluster for the development site. I've now created this account to handle link the two systems.

I'll contact you again when I'm ready to migrate the production site.

Please let me know if you have any questions.

Sincerely,

-Francisco Lopez


On Wed, May 17, 2017 at 10:25 AM, <hpc-support@uci.edu> wrote:

   Greetings.
   This email was sent to notify you that your HPC Cluster account is ready.
   It is mandatory that ALL NEW HPC users read the following HPC web page before
   running jobs:
           http://hpc.oit.uci.edu/running-jobs


   Due to security issues, the cluster can only be accessed from a machine on the
   UCI network:  VPN or on-campus machines on the .uci.edu domain.
   You will need to use SSH or an equivalent secure login method such as x2go to
   access the cluster.
   The HPC Cluster name is: hpc.oit.uci.edu
   To login using ssh:
           ssh  drwhite@hpc.oit.uci.edu
   To enable X11 ( if you want GUI to display ):
           ssh -X -Y drwhite@hpc.oit.uci.edu
   Your password is your UCInetID password.  If you forgot your UCInetID password,
   you can reset at:  http://www.oit.uci.edu/ucinetid/ucinetid-password.html


   HPC Main web page can be found at:
            http://hpc.oit.uci.edu
   To see what queues ( private / public ) you have access to, enter:
           queue
   at the system prompt.   You can also use 'q' for short.


   If you do not have a secure method of connecting to HPC, we recommend using
   x2go.  You may download your x2go client from:
          http://www.x2go.org/doku.php/download:start
   If you are having issues accessing the HPC Cluster, please email the HPC
   support team at hpc-support@uci.edu
   Note: It may take up to 5 minutes for your account to be activated.
   Welcome to the UCI campus High Performance Cluster,
   The HPC Team
   _______________________________________________
   hpc-support mailing list
   hpc-support@uci.edu
   https://maillists.uci.edu/mailman/listinfo/hpc-support

HPC 2017 May 17

Greetings.

This email was sent to notify you that your HPC Cluster account is ready.

It is mandatory that ALL NEW HPC users read the following HPC web page before running jobs:

       http://hpc.oit.uci.edu/running-jobs   HPC

Due to security issues, the cluster can only be accessed from a machine on the UCI network: VPN or on-campus machines on the .uci.edu domain.

You will need to use SSH or an equivalent secure login method such as x2go to access the cluster.

The HPC Cluster name is: hpc.oit.uci.edu

To login using ssh:

       ssh  drwhite@hpc.oit.uci.edu

To enable X11 ( if you want GUI to display ):

       ssh -X -Y drwhite@hpc.oit.uci.edu

Your password is your UCInetID password. If you forgot your UCInetID password, you can reset at: http://www.oit.uci.edu/ucinetid/ucinetid-password.html


HPC Main web page can be found at:

        http://hpc.oit.uci.edu

To see what queues ( private / public ) you have access to, enter:

       queue

at the system prompt. You can also use 'q' for short.


If you do not have a secure method of connecting to HPC, we recommend using x2go. You may download your x2go client from:

      http://www.x2go.org/doku.php/download:start

If you are having issues accessing the HPC Cluster, please email the HPC support team at hpc-support@uci.edu

Note: It may take up to 5 minutes for your account to be activated.

Welcome to the UCI campus High Performance Cluster, The HPC Team

April 18 2017

Hi Doug,

We're using the Torque scheduler at UCI; XSEDE is using Slurm. I think it's a simple matter to switch Galaxy from using one or the other. I'll talk with Harry about setting up an appropriate account and test switching to UCI's HPC cluster.

Sincerely,

-Francisco

April 15 2017

To: Francisco Lopez

can we talk about this? i.e., a working http://socscicompute.ss.uci.edu ? So far, I did a renewal with http://portal.xsede.org. Thanks Doug


Forwarded Message --------

Subject: Re: my http://socscicompute.ss.uci.edu got derailed during my bout with cancer Date: Thu, 13 Apr 2017 01:00:15 +0000 From: Wilkins-Diehr, Nancy <wilkinsn@sdsc.edu> To: Doug White <douglas.white@uci.edu>, Ken Hackworth <hackwort@psc.edu>

       Doug,
First step, let’s get an active Comet allocation. You do that at http://portal.xsede.org. I thought you already went into that, which is where you saw that you could only renew for 6 months. Let’s do that and at least get that far.
If I were you I would call Francisco from UCI on the phone and talk through the variables on the SocSci form and see if he can help debug that aspect. You and he could use Google hangouts and screen share to get to the root of the problem more quickly than with email.

Nancy

From: Doug White <douglas.white@uci.edu> Date: Wednesday, April 12, 2017 at 5:40 PM To: Ken Hackworth <hackwort@psc.edu> Subject: my http://socscicompute.ss.uci.edu got derailed during my bout with cancer

One can still click on the 2rd,3rd,4th boxes with numeric variables starting eg with one var eg v5 in the 2nd row and three or more variables in the 3rd,4th row vars but the old system doesnt work -- it stopped working while I had the cancer. Any suggestion?  At http://socscicompute.ss.uci.edu essentially its a very simple matter of entering these kinds of numbers.

I suspected that if I get a new allocation on Comet, which I did at SDSC, it would be a very simple fix to have my http://socscicompute.ss.uci.edu work again (it ran out of time when I had cancer last year). I did get a new allocation on Comet but dont know what to do next or who to email for help. Question: where do I go for the renewal request?

Many thanks, sorry for the bother.

I am at UC Irvine as is http://socscicompute.ss.uci.edu but it originates at SDSC.

Nancy says: Doug, After consulting with Ken (the expert), we believe all you need to do is submit a renewal request.

Hi Doug,

  • The VM farm that Harry is referring to is a great place to have a server. It is a very reliable virtual machine infrastructure similar to AWS, but hosted exclusively on-campus in UCI OIT's data center.
  • I'm not quite sure what it is you would like to setup... you mention "DEf01f" which is one of the links on the left side bar of your socscicompute.ss.uci.edu galaxy instance. But I'm not sure what you mean by "cross-cultural analyses". If you have a new tool or workflow that you'd like setup on your socscicompute.ss.uci.edu server, then you'd have to contract with Francisco to get that done.
  • Best,
  • Jonathan
  • Harry Mangalam -- where CoSSci is located
  • Francisco Lopez <lopez@uci.edu> 1 949 824-8818

12th Jan 2017

Hi Doug,

I've reviewed the information. I can resume migrating the content on http://socscicompute.ss.uci.edu to a virtual server running in Amazon's Web Services; I just wanted to confirm that this is what you had in mind.

Sincerely,

-Francisco

Nancy

Sounds good!

New Year, early 2017

Hi, good to talk to you, Francisco, as exemplified here

http://socscicompute.ss.uci.edu/ which was once a live means of accessing a specific database system

where one can click to begin an analysis starting with

DEf01f Dow Eff

that allows use the database for regession analysis all of which I was trying to port to what was being discussed before I got sick. Won't bother Harry as yet but will be trying to map out what I was saying belo.

Here are more notes before I got stomach cancer in fall of 2015: click http://intersci.ss.uci.edu/wiki/index.php/Francisco_Lopez

- similar notes to click in discussion http://intersci.ss.uci.edu/wiki/index.php/Harry_Mangalam

prior to fall 2015

Hi Francisco:

I would prefer option 1 to minimize our administrative responsibility for the server.

I am planning to use Galaxy. Details of standalone setup, and under Apache, are here:

http://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer
http://wiki.galaxyproject.org/Admin/Config/Apache%20Proxy

A couple questions:

- Will I have full filesystem access to the application directory used by Apache? If so, I can set up Galaxy once you've set up the related Apache bits. - Do you run a database server where you could host a database for us? If not, I could start with just the Sqlite support for now.

We'll agree on a name to use for the virtual server and get back to you with that soon.

Thanks Francisco,

Earlier

Tom Uram (Senior Software Developer turam@mcs.anl.gov who helped set up "galaxy" software for an initial interface in 2010 that was working in 2009-2015)

Later, 2012 can be reinvigorated in 2107

On Dec 10, 2012, at 7:47 PM, Francisco Lopez wrote: (THIS IS STILL PERTINENT FOR 2017- DRW)

> Hi Thomas, > > We actually have two options.

  • 1. Dr. White and I discussed placing the portal inside an Apache virtual host. You would then be a user on a shared Apache web server. We run Red Hat Enterprise Linux 6 with Python 2.6.6.
  • 2. Another option would be to create a VM host for your exclusive use. In this scenario, you can install Ubuntu and would have root access; this also means you'll have full responsibility for patching and maintaining the OS.
  • Both options cost roughly the same. Let me know if you have a preference between the two options.
  • Sincerely,
  • -Francisco

on 12/6/12 2:04 PM Thomas Uram said the following:

Hi Francisco:

The OS version should not be important as long as its recent. I'd prefer Ubuntu if there's a choice.

I'll be using Python, and only require that it also be a recent version in the 2.x line.

PHP will not be used.

How flexible is the environment? Is this a controlled environment in which I am a user? Or is it a VM to which I might have root access? I don't require root access, but simply want to know what to expect.

Thanks, Tom


On Dec 5, 2012, at 2:32 PM, Francisco Lopez wrote:

 Hi Thomas,

I've spoken with Dr. White about creating a web site to host portal software that you will be installing. Please let me know the technical requirements of the software, i.e. OS version, PHP version, etc.

Let me know if you'd like to schedule a phone call to discuss the details.

Sincerely,

 -Francisco

 --
 Francisco Lopez
 Office of Information Technology (OIT)
 University of California, Irvine
 949-824-8818

Harry Mangalam

Hi Doug,

Got your voicemail. Very glad to hear about your cancer being in remission.

The 'location' of your server is that it's a Virtual Machine on one of UCI's VM farms.

Francisco Lopez can give you more information should you need it re: size/OS/physical location of the VM farm

Virtual Machine on one of UCI's VM farms size/OS/physical location of the VM farm