diff --git a/README.rst b/README.rst
index 9d22160f9064b7a9d114bf918f7fb614a6e1c4f6..171c84db91bb78dc46f86a293fc0f3ba5f411195 100644
--- a/README.rst
+++ b/README.rst
@@ -1,47 +1,9 @@
Autotuning and Predictive Autotuning
====================================
-``predtuner`` performs autotuning on program approximation knobs using an error-predictive proxy
+PredTuner performs autotuning on program approximation knobs using an error-predictive proxy
in place of the original program, to greatly speedup autotuning while getting results
comparable in quality. ``current_version == 0.3``.
-Installation
-------------
-
-``predtuner`` requires ``python >= 3.6`` and ``pip``, preferrably ``pip >= 20``.
-To install this package from source, at the root directory of this repository, do:
-
-.. code-block:: bash
-
- python3 -m pip install -e ./
-
-With the flag ``-e``, any changes to code in this repo is reflected on the installed version automatically.
-It can be omitted if you don't intend to modify the code in this package.
-
-Getting Started
----------------
-
-The documentation page contains a full tutorial.
-Build the documentation by:
-
-.. code-block:: bash
-
- pip install sphinx sphinx_rtd_theme sphinx_autodoc_typehints
- cd doc
- make html
-
-The documentation page will be created as ``doc/build/html/index.html``.
-You can open this in the browser and browse to "Getting Started" section.
-
-Model Data for Example / Testing
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-``predtuner`` contains 10 demo models which are also used in tests.
-
-
-* Download and extract `this <https://drive.google.com/file/d/1V_yd9sKcZQ7zhnO5YhRpOsaBPLEEvM9u/view?usp=sharing>`_ file containing all 10 models, for testing purposes.
-* The "Getting Started" example on the documentation page only uses VGG16-CIFAR10.
- If you don't need the other models, get the data for VGG16-CIFAR10
- `here <https://drive.google.com/file/d/1Z84z-nsv_nbrr8t9i28UoxSJg-Sd_Ddu/view?usp=sharing>`_.
-
-In either case, there should be a ``model_params/`` folder at the root of repo after extraction.
+Read our `documentation here <https://predtuner.readthedocs.io/en/latest/index.html>`_
+for how to install and use PredTuner.
diff --git a/doc/_static/result_no_model.png b/doc/_static/result_no_model.png
new file mode 100644
index 0000000000000000000000000000000000000000..6f29f7d6bc8aa8237cd0588506d4eb7a285a9d9a
Binary files /dev/null and b/doc/_static/result_no_model.png differ
diff --git a/doc/_static/result_with_model.png b/doc/_static/result_with_model.png
new file mode 100644
index 0000000000000000000000000000000000000000..095b6b76981e230a7d60c16f31928e56baaeeef6
Binary files /dev/null and b/doc/_static/result_with_model.png differ
diff --git a/doc/conf.py b/doc/conf.py
index 8fe20cee8424b1d12a07e7426885920c3dad13da..3a97e1483cb8b0a91ebdfaafd15e6858c0e2f01e 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -75,7 +75,7 @@ html_theme_options = {
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = []
+html_static_path = ["_static"]
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
diff --git a/doc/getting_started.rst b/doc/getting_started.rst
index 5d8472a5a224b6677df4f080bf6309bff6bae016..d0e1305cb601f9ae6a2b26d869e38091851302e2 100644
--- a/doc/getting_started.rst
+++ b/doc/getting_started.rst
@@ -6,24 +6,43 @@ This guide can help you start working with PredTuner.
Installation
------------
-Install PredTuner from source using `pip`:
+* PredTuner requires ``python >= 3.6`` and ``pip``, preferrably ``pip >= 20``.
+
+To install this package from source, at the root directory of this repository, do:
.. code-block:: shell
- pip install -e .
+ python3 -m pip install -e ./
PredTuner will also be available on PyPi in the future after we publish the first release.
+* With the flag ``-e``, any changes to code in this repo is reflected on the installed version automatically.
+ It can be omitted if you don't intend to modify the code in this package.
+
+Model Data for Example / Testing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+PredTuner contains 10 demo models which are also used in tests.
+
+* Download and extract `this <https://drive.google.com/file/d/1V_yd9sKcZQ7zhnO5YhRpOsaBPLEEvM9u/view?usp=sharing>`_ file containing all 10 models, for testing purposes.
+* In the tutorial below, we will only use VGG16-CIFAR10.
+ If you don't need the other models, get the data for VGG16-CIFAR10
+ `here <https://drive.google.com/file/d/1Z84z-nsv_nbrr8t9i28UoxSJg-Sd_Ddu/view?usp=sharing>`_.
+
+In either case, there should be a ``model_params/`` folder at the root of repo after extraction.
+
Tuning a PyTorch DNN
--------------------
+* The code used in the following example can be found at ``examples/tune_vgg16_cifar10.py``.
+
PredTuner can tune any user-defined application,
but it is optimized for tuning DNN applications defined in PyTorch.
We will use models predefined in PredTuner for demonstration purposes.
Download pretrained VGG16 model parameters and CIFAR10 dataset from `here
<https://drive.google.com/file/d/1Z84z-nsv_nbrr8t9i28UoxSJg-Sd_Ddu/view?usp=sharing>`_.
-After extraction, there should be a `model_params/` folder in current directory.
+After extraction, there should be a ``model_params/`` folder in current directory.
Load the tuning and test subsets of CIFAR10 dataset, and create a pretrained VGG16 model:
@@ -55,7 +74,7 @@ while the test dataset is used to evaluate configurations found in autotuning.
This is similar to the split between training and validation set in machine learning tasks.
In this case, both tuning and test datasets contain 5000 images.
-Create an instance of `TorchApp` for tuning PyTorch DNN:
+Create an instance of `~predtuner.torchapp.TorchApp` for tuning PyTorch DNN:
.. code-block:: python
@@ -69,31 +88,33 @@ Create an instance of `TorchApp` for tuning PyTorch DNN:
model_storage_folder="vgg16_cifar10/",
)
-PredTuner provides `TorchApp`, which is specialized for the use scenario of tuning PyTorch DNNs.
+PredTuner provides `~predtuner.torchapp.TorchApp`,
+which is specialized for the use scenario of tuning PyTorch DNNs.
In addition, two more functions from PredTuner are used:
-`pt.accuracy` is the *classification accuracy* metric,
+:py:meth:`pt.accuracy <predtuner.torchutil.accuracy>`
+is the *classification accuracy* metric,
which receives the probability distribution output from the VGG16 model,
compare it to the groundtruth in the dataset,
-and returns a scalar between 0 and 100 for the classification accuracy
+and returns a scalar between 0 and 100 for the classification accuracy.
-`pt.get_knobs_from_file()` returns a set of approximations preloaded in PredTuner,
+:py:meth:`pt.get_knobs_from_file <predtuner.approxes.get_knobs_from_file>`
+returns a set of approximations preloaded in PredTuner,
which are applied to `torch.nn.Conv2d` layers.
-See ??? for these approximations and how to define custom approximations.
Now we can obtain a tuner object from the application and start tuning.
We will keep configurations that don't exceed 3% loss of accuracy,
-but encourage the tuner to find configurations with loss of accuracy below 2.1%.
+but encourage the tuner to find configurations with loss of accuracy below 2.0%.
.. code-block:: python
tuner = app.get_tuner()
tuner.tune(
- max_iter=500,
- qos_tuner_threshold=2.1, # QoS threshold to guide tuner into
+ max_iter=1000,
+ qos_tuner_threshold=2.0, # QoS threshold to guide tuner into
qos_keep_threshold=3.0, # QoS threshold for which we actually keep the configurations
- is_threshold_relative=True, # Thresholds are relative to baseline -- baseline_acc - 2.1
- take_best_n=50,
+ is_threshold_relative=True, # Thresholds are relative to baseline -- baseline_acc - 2.0
+ take_best_n=20,
cost_model="cost_linear", # Use linear cost predictor
)
@@ -101,8 +122,11 @@ but encourage the tuner to find configurations with loss of accuracy below 2.1%.
e.g., here it refers to the accuracy of DNN over given datasets.
We will be using the term QoS throughout the tutorials.
+:py:meth:`tuner.tune <predtuner.modeledapp.ApproxModeledTuner.tune>`
+is the main method for running a tuning session.
+It accepts a few parameters which controls the behavior of tuning.
`max_iter` defines the number of iterations to use in autotuning.
-Within 500 iterations, PredTuner should find about 200 valid configurations.
+Within 1000 iterations, PredTuner should find about 200 valid configurations.
PredTuner will also automatically mark out `Pareto-optimal
<https://en.wikipedia.org/wiki/Pareto_efficiency>`_
configurations.
@@ -111,7 +135,7 @@ in contrast to "valid" configurations which are the configurations that satisfy
(`tuner.kept_configs`).
`take_best_n` allows taking some extra close-optimal configurations in addition to Pareto-optimal ones.
-500 iterations is for demonstration; in practice,
+1000 iterations is for demonstration; in practice,
at least 10000 iterations are necessary on VGG16-sized models to converge to a set of good configurations.
Depending on hardware performance, this tuning should take several minutes to several tens of minutes.
@@ -130,7 +154,8 @@ and visualize all configurations in a figure:
The generated figure should look like this:
-.. image:: tuning_result.png
+.. image:: _static/result_no_model.png
+ :target: _static/result_no_model.png
where the blue points shows the QoS and speedup of all valid configurations,
and the "best" configurations are marked out in orange.
@@ -148,11 +173,11 @@ To do that, simply use the argument `qos_model` when calling `tuner.tune()`:
tuner = app.get_tuner()
tuner.tune(
- max_iter=500,
- qos_tuner_threshold=2.1, # QoS threshold to guide tuner into
+ max_iter=1000,
+ qos_tuner_threshold=2.0, # QoS threshold to guide tuner into
qos_keep_threshold=3.0, # QoS threshold for which we actually keep the configurations
- is_threshold_relative=True, # Thresholds are relative to baseline -- baseline_acc - 2.1
- take_best_n=50,
+ is_threshold_relative=True, # Thresholds are relative to baseline -- baseline_acc - 2.0
+ take_best_n=20,
cost_model="cost_linear", # Use linear cost predictor
qos_model="qos_p1"
)
@@ -162,3 +187,17 @@ when it learns about the behavior of each knob on each operator (DNN layer).
Because the configurations will end up with predicted QoS values after tuning,
this will add a *validation* stage at the end of tuning where the QoS of best configurations are empirically measured,
and the bad ones are removed.
+
+Following the procedure above to plot a figure of the configurations,
+the generated figure should look like this,
+with one extra subfigure (middle) comparing the predicted and measured QoS.
+
+.. image:: _static/result_with_model.png
+ :target: _static/result_with_model.png
+
+----------------------------------------------------------
+
+This concludes the tutorial for installing and using PredTuner.
+What we have just used is the PyTorch API of PredTuner.
+:doc:<reference/index> shows the reference of this API along with two sets of lower-level APIs
+that allows tuning applications that are not PyTorch DNN.
diff --git a/doc/reference/index.rst b/doc/reference/index.rst
index e94a2e79da525f24a8fe700a6efb140e963ec8d8..bcffe6c75b357705c9bc85fc1daac0323282b1cc 100644
--- a/doc/reference/index.rst
+++ b/doc/reference/index.rst
@@ -3,7 +3,7 @@ PredTuner Autotuning API
:doc:`pytorch-app` documents a high-level API for autotuning PyTorch Module.
-`predtuner` also supports predictive tuning of general applications that are not PyTorch Module,
+PredTuner also supports predictive tuning of general applications that are not PyTorch Module,
or even empirical tuning of general application that doesn't support predictive models.
These lower-level APIs are documented in :doc:`modeled-app` and :doc:`approx-app` respectively.
diff --git a/doc/tuning_result.png b/doc/tuning_result.png
deleted file mode 100644
index 6210102982aefa78a796a7a8593fe766e802dbf9..0000000000000000000000000000000000000000
Binary files a/doc/tuning_result.png and /dev/null differ