Jobs with python code and subdirectories


#1

I am testing paperspace, and I am having trouble running my python code.

Locally, I have a python code in a main directory, and a lot of classes in subdirectories. I send my code using the following command:

paperspace-python run main.py --python 3 --project myproject --machineType GPU+ --container jupyter/datascience-notebook --workspace .

All files in the subdirectory are uploaded to paperspace, I can see them in the “Files” tab of the job created. Even __init__.py files are in the subdirectories. But I get the following error:

New jobId: jmf2jerhn44xc
Cluster: PS Jobs
Job Pending
Waiting for job to run...
Awaiting logs...
[WARN  tini (7)] Tini is not running as PID 1 .
Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
To fix the problem, run Tini as PID 1.
Traceback (most recent call last):
  File "main.py", line 116, in <module>
    from caju_py.news.news_objs import NewsQuery
ModuleNotFoundError: No module named 'caju_py.news'
Job Failed; exitCode 1

which seems like files in my subdirectory are not found.

Can anyone help me?


#2

This may be related to when the source files are coming from a windows machine and the line ending conversion is not set up to translate the line endings to UNIX.

Another source of the problem could be that the underlying python packages are not being installed, which is more likely. If the packages have native code dependencies then those need to be installed as part of the job initialization.

The paperspace-python docs cover many different options for doing this automatically.

A third possibility is that you may have not configured the python module path appropriately for your application. It cannot predicted ahead of time what your module path list should be, and in some cases you need to configure or augument the existing module paths of the job environment if your python package tree depends on modules that are not accessible in the tree from the default root directory of your upload to the job working directory.

Hope that helps!


#3

The paperspace-python documentation and the dependencies worked for me.

First I removed some dependencies. I changed my container to “paperspace/tensorflow-python” as well. Then I ran the job within my python code and it worked.

Thank you Stan!