Context switching for Qbox objects

[mgoldey]

Hi Everyone,

I have several embarrassingly parallel routines I could call on separate pools of processors for code on which I'm working, but it's not clear that there's a trivial way to duplicate any matrix/wavefunction objects on subsets of the main context.

One can certainly code something up to do synchronized broadcasts and receives of data (a la Matrix::getsub), but I thought this might be a common enough problem that someone has a clever solution that's more elegant than I'll come up with at first glance, especially since I'd ideally want to be able to adjust the context during runtime as the number of trivially parallelizable routines grows.

Please feel free to suggest optimal ways you've found or to encourage a different method of code organization. I'll hack through this once I have everything working in serial for my code, so if no one has any bright ideas, I'll post my solution here.

Matthew

[mgoldey]

General notes on making contexts from contexts and moving data around, stored in this thread for convenient use of others. Working toward what I wanted in the parent post.

Contexts can be constructed from other contexts, via Context.h or testContext.C

Sample code snippets below:

Converts 4x1 to 2 contexts of size 2, stored in array c

Code: Select all

Context ctxt(4,1);
ctxt.print(cout);
vector<Context*> c;
ctxt.barrier();
c.push_back(new Context(ctxt,2,1,0,0));
ctxt.barrier();
c.push_back(new Context(ctxt,2,1,2,0));

Converts 2x2 context to 2 contexts of size 2, stored in array c

Code: Select all

Context ctxt(2,2);
ctxt.print(cout);
vector<Context*> c;
ctxt.barrier();
c.push_back(new Context(ctxt,2,1,0,0));
ctxt.barrier();
c.push_back(new Context(ctxt,2,1,0,1));

I've created a new constructor for the Wavefunction class that can be passed a context and a previous wavefunction, a la Wavefunction my_wf(ctxt,wf)
This may work correctly or not.

Code: Select all

////////////////////////////////////////////////////////////////////////////////
//MBG create wavefunction in new context from another wavefunction
////////////////////////////////////////////////////////////////////////////////
Wavefunction::Wavefunction(const Context& ctxt, const Wavefunction& wf) : ctxt_(ctxt),
nel_(wf.nel_), nempty_(wf.nempty_), nspin_(wf.nspin_),
deltaspin_(wf.deltaspin_), nrowmax_(wf.nrowmax_),
cell_(wf.cell_), refcell_(wf.refcell_),
ecut_(wf.ecut_), weight_(wf.weight_), kpoint_(wf.kpoint_)
{
  // Create a Wavefunction using the dimensions of the argument

  compute_nst();

  // Next lines: do special allocation of contexts to ensure that
  // contexts are same as those of wf
  spincontext_ = new Context(ctxt_);
  kpcontext_ = new Context(ctxt_);
  sdcontext_ = new Context(ctxt_);

  allocate();

  resize(cell_,refcell_,ecut_);
  init();
}

To get a wavefunction belonging to this context, we'd use

Code: Select all

int mycol=ctxt->mycol();
Context* myc=c[mycol];
Wavefunction lwf(*myc,wf_);

Matrices can be given the desired context shape by giving additional arguments to their constructor - here we create a matrix foo in context ctxt from matrix bar with maximum column block size of 1.

Code: Select all

ComplexMatrix foo(ctxt, bar.m(),bar.n(),64,1);
foo.getsub(bar,bar.m(), bar.n(),0,0);

[mgoldey]

I'm realizing that I need to do an alltoallv broadcast to get everything on the right processors. Anyone done this successfully?

[mgoldey]

I'm finding that the wavefunction reorders g-vectors upon a change of context, making an alignment of the data difficult.

In contrast, it is fairly straightforward to dump the wavefunction to disk and read it in again.

Code: Select all

  const string filename_str="tmp_wfn";
  const char* filename_cstr=filename_str.c_str();
  SampleWriter s_writer(s_.ctxt_);
  string description = string(" Created ") + isodate() +
                       string(" by qbox-") + release() + string(" ");
  s_writer.writeSample(s_,filename_str,description,false, false, false, false);

  cout<< "Wrote wavefunction"<<endl;

To read it in again :

Code: Select all

      SampleReader s_reader(*c[i]);
      try
      {
        s_reader.readSample(*mys,filename_cstr,false);
      }
      catch ( const SampleReaderException& e )
      {
        cout << " SampleReaderException caught in LoadCmd:" << endl;
        cout << e.msg << endl;
      }
      catch (...)
      {
        cout << " LoadCmd: cannot load Sample" << endl;
      }

Unfortunately, calls to generate a new sample on a subset of processors

Code: Select all

    
Sample* mys=new Sample(*c[i]);

hang since Cblacs_gridinit invokes a subroutine with MPI_WORLD_COMM. This could be worked around by making an xml parser just for the wavefunction...