I would want to procedurally generate totally new images from a set of thousands of images of a given category (simple landscapes for example). Deep-learning won't be used for this purpose. This new image will look likes these thousands of images, but will present variations in how their edges are drawn (position, length, angles, neighbouring edges). Colorizing the output image is done by deep-learning Picture2Picture ("Pix2Pix") program trained on the same given category.
Even if we don't use machine-learning, we can split the worflow into two parts:
Workflow > Training
I have thousands of grayscaled images. For each image, I extract its edges into a graph of edges. So I end with thousands of graphs of edges.
Each graph is defined by:
- A node is a point
- A link is an edge
- A node contains these metadata: its abscissa position, its ordinate position
- So a graph contains these metadata: the positions of its points, the length of its edges, their angles, their neighboring edges.
I would want to use these thousands of graphs in order to compute a set of edges-variators (defined below). To do this, I would use a system that would take the thousands of graphs as inputs, and would output this set of edges-variators.
This set of edges-variators is defined by:
- An edges-variator contains these data: a set of abscissa positions, and a set of ordinate positions, both are of course in bijection. Definition of an edges-variator: it can output an edge by taking one abscissa position plus its corresponding ordinate position.
- Each edges-variator is associated to each edge that is contained in almost all of the thousands of graphs. It implies to be able to identify the edges that are contained in almost all of the thousands of graphs AND to consider that edges with tiny differences between their points coordinates are in fact a same edge.
- There is no edges-variator that would be associated to each edge that is contained in only some of the thousands of graphs.
- Each edges-variator has its neighboring edges-variator. It's a consequence from 2. et 3.
- How are computed the abscissa and ordinate positions sets of each edges-variator? As each edge of these edges-variator is contained in almost all of these thousands graphs, we retrieve the corresponding edge in each of these graphs, get its abscissa and ordinate positions, and use them to fill these edges-variator's sets.
Workflow > Training ends
We save this edges-variators set.
Workflow > Production
Then, we want to generate a new image: we load this edges-variators set, then we take each edges-variator, and randomly choose the coordinates of each of its both points (by picking them from its abscissa and ordinate sets). The used randomness is based on the distribution of these coordinates: we will pick-up coordinates that are often used, rather than extreme coordinates. We will add a tiny random number to each of the chosen coordinates of each point of the edge. We obtain a new graph of edges, from these computations on all the edges-variators. This graph has the same definition than the one defined in the part "Training". These edges are contained in almost all of the thousands images, but are drawn with different coordinates, lengths, angles.
Workflow > Production ends
Finally, we convert this new graph into an image by drawing the edges. We colorize this image by using deep-learning Picture2Picture ("Pix2Pix") program that must be trained in images of the same category (cf. my first sentence). Then we save this colorized image.
We could imagine to create animated images or even short videos since the new created image is drawn from a graph of edges. Idem, we can imagine to create a music from this graph of edges, if we have succeeded in creating animated or short videos (by using edges' points' coordinates).
- The training images must be of the same category, taken from about the same angle, with about the same elements placed in about the same place. NB: perhaps using deep-learning to isolate the elements, and/or the background elements from the main ones, could help here.
First of all: does it seem realizable for simple landscapes? If no: you can stop your reading :-) . If yes: please continue to read the questions.
Do you know any research paper, any Github/Gitlab MIT (or equivalent licence) that implements this idea?
Do you think the training part would be a lot faster than deep-learning GANs? What about the production part?